Quantized LLM training in pure CUDA/C++.

Separate from hardware and used to learn some NCCL mechanisms

An effort to "ssh into my Sony camera"

Reduce kernel based on CUTLASS CuTe and TMA.

Code for the 9/6 Hackathon

RAM is all you need

AI-powered dialogue generation for Animal Crossing villagers using LLMs

Efficient Transformers computation kernels lowering through MLIR

🔀 yet another mixture of experts

A minimal tensor processing unit (TPU), inspired by Google's TPU V2 and V1

A Quirky Assortment of CuTe Kernels

Simple MPI implementation for prototyping or learning

Experimental repository for research implementation of NoLoCo.

AXI, AXI stream, Ethernet, and PCIe components in System Verilog

Research sandbox for decentralized pipelined inference

Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse environments

Async RL Training at Scale

Scripts and instructions for replicating the original FineWeb experiments on LUMI

Muon fsdp 2

Analyze computation-communication overlap in V3/R1.

A bidirectional pipeline parallelism algorithm for computation-communication overlap in DeepSeek V3/R1 training.

Expert Parallelism Load Balancer

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

Where GPUs get cooked 👩‍🍳🔥

DeepEP: an efficient expert-parallel communication library