Ongoing research training transformer models at scale

Scalable toolkit for efficient model reinforcement

Training library for Megatron-based models

A high-throughput and memory-efficient inference and serving engine for LLMs

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilizatio…

cudnn_frontend provides a c++ wrapper for the cudnn backend API and samples on how to use it

DeepEP: an efficient expert-parallel communication library

A tool to configure, launch and manage your machine learning experiments.

Scalable toolkit for efficient model alignment

NeMo Megatron launcher and tools

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.