What's Changed

• Add Qwen3 2507 Thinking models by @ysjprojects in #2110
• Add MLA by @ysjprojects in #2113
• Add Qwen3 2507 Instruct models by @ysjprojects in #2125
• finetune_lora upgrades by @ysjprojects in #2086
• Use torch autograd integration for backward testing by @t-vi in #2123
• bump PT 2.8 & install with --upgrade-strategy greedy by @Borda in #2107
• build(deps): update transformers requirement from <4.54,>=4.51.3 to >=4.51.3,<4.57 by @dependabot[bot] in #2120
• build(deps): update huggingface-hub requirement from <0.33,>=0.30 to >=0.30,<0.35 by @dependabot[bot] in #2119
• build(deps): update bitsandbytes requirement from <0.43,>=0.42 to >=0.42,<0.47 by @dependabot[bot] in #2069

Full Changelog: v0.5.10...v0.5.11

What's Changed

• qwen2.5 long context by @ysjprojects in #1933
• debug installing torch for Thunder by @Borda in #2072
• Fix in convert_hf_checkpoint related to Gemma 3 by @mseeger in #2062
• debug some failing standalone tests with compiler by @Borda in #2075
• Qwen3 MoE by @ysjprojects in #2060
• Deferring import of torch in config to allow faster import by @JackUrb in #2079
• update bug-report/issue with reproducing in Studio by @Borda in #2081
• ci: show the longest tests for improvement by @Borda in #2083
• docs: Add documentation for OpenAI-compatible API in LitGPT deployment by @bhimrazy in #2082
• doc: add n_query_groups to attention notation table by @raishish in #2092
• doc: add comments for clarifying query / KV groups by @raishish in #2093
• Complete pending todos in testing by @raishish in #2088
• pin: restrict datasets version to <4.0.0 for compatibility by @Borda in #2095
• add/debug Lit CI by @Borda in #2094
• add api_path parameter to cli options to allow custom API endpoint configuration by @botirk38 in #2080
• Fix commands in pretrain.md by @H4kor in #2097

New Contributors

• @JackUrb made their first contribution in #2079
• @raishish made their first contribution in #2092
• @botirk38 made their first contribution in #2080
• @H4kor made their first contribution in #2097

Full Changelog: v0.5.9...v0.5.10

What's Changed

• input_pos_maxp1 as a Python integer by @Andrei-Aksionov in #2016
• add testing for py3.12 & py3.13 by @Borda in #2025
• ci: update guardian for PRs by @Borda in #2043
• Add devcontainer by @twsl in #2035
• Remove dependency from config to utils by @lukemerrick in #2034
• fix: Add fallback chat template by @andyland in #2040
• Add optional sys prompt by @twsl in #2036
• fix: Pretraining text files with recent litdata versions by @andyland in #2048
• tests: mark test_evaluate_script as flaky by @Borda in #2049
• simplify the GPU testing flow by @Borda in #2053
• ci: extend testing with ubuntu-24.04 by @Borda in #2056
• Remove litserve version constraint by @twsl in #2055
• req: pin bitsandbytes>=0.45.2,<0.45.5 by @Borda in #2057
• ci: use Thunder dev images for testing by @Borda in #2054
• Update spacing in README.md by @Borda in #2058
• Transformers version bump by @KaelanDt in #2029
• Qwen3 Dense by @ysjprojects in #2044
• phi-4 reasoning models by @ysjprojects in #2047
• adding logger args by @ysjprojects in #1973
• Qwen3 MoE Preliminary: add intermediate_size argument to MLP modules by @ysjprojects in #2046
• OLMo 2 by @ysjprojects in #1897
• bump: testing with PT 2.7.1 by @Borda in #2063
• Add Dependabot for Pip & GitHub Actions by @Borda in #2066
• build(deps): bump litdata from 0.2.45 to 0.2.49 by @dependabot in #2068

New Contributors

• @twsl made their first contribution in #2035
• @lukemerrick made their first contribution in #2034
• @prabhuteja12 made their first contribution in #2027
• @andyland made their first contribution in #2040
• @KaelanDt made their first contribution in #2029

Full Changelog: v0.5.8...v0.5.9

Many great updates!

What's Changed

• add missing r1 prompt style by @ali-alshaar7 in #1929
• fix incremental save for PyTorch 2.6 by @t-vi in #1928
• fix: resolve failing CI by @Borda in #1944
• handle wrapped thundermodules in generate by @t-vi in #1955
• fix skip condition by @t-vi in #1956
• ci: use HF cache by @Borda in #1958
• ci: split HF caching by @Borda in #1960
• bump: PT 2.6 + bitsandbytes & standalone tests by @Borda in #1959
• prune whitespaces for code readability by @Borda in #1962
• fixing various typos in examples & tutorials by @Borda in #1963
• fix n_query_groups for llama-3.1-405b by @ysjprojects in #1946
• tests: make flaky test due to connection issues by @Borda in #1964
• Fix: incorrect gradient accumulation steps bug by @ysjprojects in #1947
• fix: use default num_nodes=1 for back-compatibility by @Borda in #1967
• Do not wrap LoRA layers with FSDP by @janEbert in #1538
• Speculative decoding: Base implementation by @Andrei-Aksionov in #1938
• Better clarity on SFT dataset attributes by @ysjprojects in #1970
• Enforce Consistent Formatting and Validation for YAML Files by @Borda in #1977
• Apply Standard Formatting and Fix Import & Test Name Issues by @Borda in #1981
• Setting config.sliding_window_layer_stride explicity by @ysjprojects in #1972
• feat: add linear rope type by @k223kim in #1982
• feat: update tests for transformers 4.50.2 by @k223kim in #1983
• fix: test_tokenizer_against_hf by @Borda in #1984
• feat: replace sliding window type with offset by @k223kim in #1989
• ci: with pull_request_target by @Borda in #1992
• Phi4 mini by @ysjprojects in #1949
• aggregate val_loss by @ysjprojects in #1971
• feat: add local base freq for rope by @k223kim in #1993
• test: flexible wait for serve start by @Borda in #1996
• fix: replace sliding window configuration parameters to sliding windows indices by @k223kim in #1995
• QwQ-32B by @ysjprojects in #1952
• feat: run thunder tests as part of LitGPT CI by @deependujha in #1975
• try pyupgrade-up py38 by @Borda in #1999
• [1/4] feat: add gemma 3 27b by @k223kim in #1998
• [2/4] add gemma 3 1b by @k223kim in #2000
• [3/4] feat: add gemma 3 4b by @k223kim in #2001
• feat: add gemma-3-12b by @k223kim in #2002
• Add resume for adapter_v2, enable continued finetuning for adapter by @altria-zewei-wang in #1354
• Fix/loading gemma 3 1b by @pquadri in #2004
• feat: add gemma 3 in readme and tutorials by @k223kim in #2005
• add borda as codeowner by @t-vi in #2007
• example for full finetuning with python code by @astrobdr in #1331
• feat: add tests for gemma3 by @k223kim in #2006
• building tutorials as mkdocs pages by @Borda in #2011
• Add mlflow logger support by @topikachu in #1985
• fix support for litserve>0.2.4 by @ali-alshaar7 in #1994
• Cast tensors in KVCache only when needed by @Andrei-Aksionov in #2017
• feat: load only text weights from multimodal gemma by @pquadri in #2008
• Feature: Adds support for OpenAISpec in litgpt serve by @bhimrazy in #1943
• drop upper bounds in dependencies by @t-vi in #2022

New Contributors

• @syntheticgio made their first contribution in #1939
• @deependujha made their first contribution in #1975
• @altria-zewei-wang made their first contribution in #1354
• @pquadri made their first contribution in #2004
• @astrobdr made their first contribution in #1331
• @topikachu made their first contribution in #1985
• @bhimrazy made their first contribution in #1943
• @Lynsoo made their first contribution in #2018

Full Changelog: v0.5.7...v0.5.8

What's Changed

• Add Deepseek r1 distill llama models by @ali-alshaar7 in #1922

New Contributors

• @ali-alshaar7 made their first contribution in #1922

Full Changelog: v0.5.6...v0.5.7

What has changed

• Phi-4 by @ysjprojects
• Thunder compatibility by @t-vi

What's Changed

• Post-release setup for 0.5.5.dev1 by @Andrei-Aksionov in #1885
• Falcon3 by @ysjprojects in #1881
• ChatML prompt template by @ysjprojects in #1882
• Small fixes and refactoring by @mseeger in #1861
• Drop interleave placement in QKV matrix by @Andrei-Aksionov in #1013
• Bump PyTorch, PyTorch-Lightning and BnB versions by @Andrei-Aksionov in #1893
• Pin version of mistune in check links workflow by @Andrei-Aksionov in #1895
• Skip converting .safetensors to .bin by @ysjprojects in #1853
• Some improvements for KV caching by @mseeger in #1891
• added query-key norm to accomodate OLMo2 by @ysjprojects in #1894
• Improve HF download speed by @rasbt in #1899
• Bump version for 0.5.5 release by @rasbt in #1901

New Contributors

• @mseeger made their first contribution in #1861

Full Changelog: v0.5.4...v0.5.5

What's Changed

• 0.5.3 post release setup by @rasbt in #1817
• Add cff file by @rasbt in #1818
• Deprecate Support for Dolly, Nous-Hermes, Redpajama-Incite, Vicuna and H2O Danube Models. by @ParagEkbote in #1821
• Adding OLMo by @aflah02 in #1827
• Adding Qwen2.5 by @ysjprojects in #1834
• Restore SlimPajama preprocessing code by @aflah02 in #1840
• Add QwQ-32B-Preview by @ysjprojects in #1844
• Add Mixtral-8x22B by @ysjprojects in #1845
• add Llama-3.3-70B-Instruct by @ysjprojects in #1859
• add Salamandra by @ysjprojects in #1857
• Qwen2.5: fix block size for Coder series by @ysjprojects in #1856
• fix: add missing"," by @vra in #1855
• fix llama3.3 readme url by @ysjprojects in #1862
• Set torch.load(..., weights_only=False) in litgpt/api.py by @Andrei-Aksionov in #1874
• Add Qwen2.5 math by @ysjprojects in #1863
• Add SmolLM2 by @ysjprojects in #1848
• Add Mistral-Large-Instruct-2411 by @ysjprojects in #1876
• Bump version for 0.5.4 release by @Andrei-Aksionov in #1883
• Temporary remove Thunder to make a release by @Andrei-Aksionov in #1884

New Contributors

• @ParagEkbote made their first contribution in #1821
• @ysjprojects made their first contribution in #1834
• @vra made their first contribution in #1855

Full Changelog: v0.5.3...v0.5.4

What's Changed

• Post-release setup for 0.5.3.dev1 by @rasbt in #1799
• Add Phi 3 128k model by @deveworld in #1800
• Add token counts to compute performance by @rasbt in #1801
• Fixed the issue that precision is always "32-true". by @jianpingw in #1802
• Add Nvidia Llama 3.1 70B Nemotron weights by @rasbt in #1803
• Choose evaluation example from test set by @rasbt in #1804
• Pretrain tok sec by @rasbt in #1805
• typo in convert_to_litgpt command by @wasifferoze in #1807
• Move distributed all_reduce import into a function by @IvanYashchuk in #1810
• Remove hardcoded 32-precision conversion by @rasbt in #1814

New Contributors

• @deveworld made their first contribution in #1800
• @jianpingw made their first contribution in #1802
• @wasifferoze made their first contribution in #1807
• @IvanYashchuk made their first contribution in #1810

Full Changelog: v0.5.2...v0.5.3

What's Changed

• Fix prompt style usage in evaluation example by @rasbt in #1790
• Pin to PyTorch 2.4.1 by @rasbt in #1796
• Fix step iteration bug in finetuning scripts by @rasbt in #1794
• Housekeeping: remove unused code and fix tests by @rasbt in #1795

Full Changelog: v0.5.1...v0.5.2