Calculate perplexity on a text with pre-trained language models. Support MLM (eg. DeBERTa), recurrent LM (eg. GPT3), and encoder-decoder LM (eg. Flan-T5).

A collection of small corpuses of interesting data for the creation of bots and similar stuff.

Crawl BookCorpus

Easily fine tune GPT-2 to fill in missing text

Front-end for ChatEval platform. Written using React, Next.js and ES6 JavaScript.

Python Implementation of HLL-tailcut with 4-bit buckets and MLE estimator

Public evaluation tool for non task driven neural open domain chatbots

A fast, efficient universal vector embedding utility package.

This repo contains code to process a once live-streamed video and annotate it with Tweets.

Deep network that performs spectral clustering

Cluster paraphrases by word sense

LogLog space version of MinHash by combining ideas from HyperLogLog and b-bit MinHash

simple python3 module for scraping google images across many languages, based on input dictionaries