Latency and Memory Analysis of Transformer Models for Training and Inference

Ongoing research training transformer language models at scale, including: BERT & GPT-2

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Example models using DeepSpeed

Source code for SIGMOD 2020 paper "Improving Approximate Nearest Neighbor Search through Learned Adaptive Early Termination"