A PyTorch Native LLM Training Framework

Decentralized RL Training at Scale

A native PyTorch Library for large model training

Collection of scripts and notebooks for OpenAI's latest GPT OSS models

Load compute kernels from the Hub

Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

👷 Build compute kernels

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

A Quirky Assortment of CuTe Kernels

Tensors and Dynamic neural networks in Python with strong GPU acceleration

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

prime is a framework for efficient, globally distributed training of AI models over the internet.

A bidirectional pipeline parallelism algorithm for computation-communication overlap in V3/R1 training.

Optimized primitives for collective multi-GPU communication

Minimalistic 4D-parallelism distributed training framework for education purpose

Ongoing research training transformer models at scale

NCCL Tests

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

A Nintendo DS emulator written in Rust for desktop devices and the web, with debugging features and a focus on accuracy

Demonstrate throughput of PyTorch FSDP

Minimal implementation of scalable rectified flow transformers, based on SD3's approach

Making large AI models cheaper, faster and more accessible

torch implementation of diloco

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.