A high-throughput and memory-efficient inference and serving engine for LLMs

A library for building fast, reliable and evolvable network services.

CUDA Library Samples

Nano vLLM

DeepEP: an efficient expert-parallel communication library

Safe rust wrapper around CUDA toolkit

Ongoing research training transformer models at scale

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

AWS Deep Learning Containers are pre-built Docker images that make it easier to run popular deep learning frameworks and tools on AWS.

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

A Python wrapper around HuggingFace's TGI (text-generation-inference) and TEI (text-embedding-inference) servers.

《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合中国宝宝的部署教程

欢迎来到 LLM-Dojo，这里是一个开源大模型学习场所，使用简洁且易阅读的代码构建模型训练框架(支持各种主流模型如Qwen、Llama、GLM等等)、RLHF框架(DPO/CPO/KTO/PPO)等各种功能。👩‍🎓👨‍🎓

Kubernetes-native Job Queueing

Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、Llama、Qwen、Baichuan、ChatGLM2、InternLM、Ziya2、Vicuna、Bloom等大模型

LeaderWorkerSet: An API for deploying a group of pods as a unit of replication

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting yo…

Following the same workflows as Kubernetes.

☸️ Effortlessly operating LLMs on Kubernetes, e.g. Serving.

A starter project with Golang, Gin and DynamoDB

A playbook for systematically maximizing the performance of deep learning models.

Kubernetes Operator for MPI-based applications (distributed training, HPC, etc.)

The fastest way to deploy a restful api's with Gin Framework with a structured project that defaults to PostgreSQL database and JWT authentication middleware stored in Redis

Install Kubernetes/K3s only, both Kubernetes/K3s and KubeSphere, and related cloud-native add-ons, it supports all-in-one, multi-node, and HA 🔥 ⎈ 🐳

The simulator for the Kubernetes scheduler

Kubernetes operator to manage installation, updation and uninstallation of tektoncd projects (pipeline, …)

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch

Golang package for gossip based membership and failure detection