Ongoing research training transformer language models at scale, including: BERT & GPT-2

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilizatio…

Tensors and Dynamic neural networks in Python with strong GPU acceleration

NeMo: a toolkit for conversational AI

PyTorch bindings for CUTLASS grouped GEMM.

Convolutions for Sequence Modeling

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

You like pytorch? You like micrograd? You love tinygrad! ❤️

JAX/Flax implementation of the Hyena Hierarchy

Ongoing research training transformer language models at scale, including: BERT & GPT-2

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

RWKV is a RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, …

Core communication lib for Bagua.

GLM-130B: An Open Bilingual Pre-Trained Model

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

An automated parallel training system that combines the advantages from both data and model parallelism. If you have any interests, please visit/star/fork https://github.com/Youhe-Jiang/OptimalShar…

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

A Massively Parallel Large Scale Self-Play Framework