Rust Client for Apache Fluss (Incubating)

verl: Volcano Engine Reinforcement Learning for LLMs

The best ChatGPT that $100 can buy.

A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

A Datacenter Scale Distributed Inference Serving Framework

A high-throughput and memory-efficient inference and serving engine for LLMs

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorR…

Nano vLLM

Ultra and Unified CCL

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Safe interop between Rust and C++

Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic.

Tensors and Dynamic neural networks in Python with strong GPU acceleration

slime is a LLM post-training framework aiming at scaling RL.

Rust RPC framework with high-performance and strong-extensibility for building micro-services.

OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient.

Apache OpenDAL: One Layer, All Storage.

SGLang is a fast serving framework for large language models and vision language models.

Distributed data engine for Python/SQL designed for the cloud, powered by Rust

Safe rust wrapper around CUDA toolkit

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

NVIDIA Inference Xfer Library (NIXL)

Cost-efficient and pluggable Infrastructure components for GenAI inference

A library that provides an embeddable, persistent key-value store for fast storage.

Efficient and easy multi-instance LLM serving

LLM inference in C/C++