A high-throughput and memory-efficient inference and serving engine for LLMs

A framework for efficient model inference with omni-modality models

llm-d is a Kubernetes-native high-performance distributed LLM inference framework

NVIDIA Inference Xfer Library (NIXL)

AI Tensor Engine for ROCm

pdb++, a drop-in replacement for pdb (the Python debugger)

Release tooling for OpenShift

vLLM adapter for a TGIS-compatible gRPC server.

Development fork of https://github.com/huggingface/text-generation-inference

Collection of Architectural Decision Records

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

[CVPR2023] CompletionFormer: Depth Completion with Convolutions and Vision Transformers

Provides an interface layer to convert between n-dimensional types in different Rust crates

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

An open source ChatGPT UI.

Examples for using ONNX Runtime for machine learning inferencing.

⚡ Building applications with LLMs through composability ⚡

Stable Diffusion web UI

Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models

Python tools for geographic data

Avalanche: an End-to-End Library for Continual Learning.