v2.2.0

Notable changes

• Llama 3.1 support (including 405B, FP8 support in a lot of mixed configurations, FP8, AWQ, GPTQ, FP8+FP16).
• Gemma2 softcap support
• Deepseek v2 support.
• Lots of internal reworks/cleanup (allowing for cool features)
• Lots of AWQ/GPTQ work with marlin kernels (everything should be faster by default)
• Flash decoding support (FLASH_DECODING=1 environment variables which will probably enable some nice improvements in the future)

What's Changed

• Preparing patch release. by @Narsil in #2186
• Adding "longrope" for Phi-3 (#2172) by @amihalik in #2179
• Refactor dead code - Removing all flash_xxx.py files. by @Narsil in #2166
• Fix Starcoder2 after refactor by @danieldk in #2189
• GPTQ CI improvements by @danieldk in #2151
• Consistently take prefix in model constructors by @danieldk in #2191
• fix dbrx & opt model prefix bug by @icyxp in #2201
• hotfix: Fix number of KV heads by @danieldk in #2202
• Fix incorrect cache allocation with multi-query by @danieldk in #2203
• Falcon/DBRX: get correct number of key-value heads by @danieldk in #2205
• add doc for intel gpus by @sywangyi in #2181
• fix: python deserialization by @jaluma in #2178
• update to metrics 0.23.0 or could work with metrics-exporter-promethe… by @sywangyi in #2190
• feat: use model name as adapter id in chat endpoints by @drbh in #2128
• Fix nccl regression on PyTorch 2.3 upgrade by @fxmarty in #2099
• Fix buildx cache + change runner type by @glegendre01 in #2176
• Fixed README ToC by @vinkamath in #2196
• Updating the self check by @Narsil in #2209
• Move quantized weight handling out of the Weights class by @danieldk in #2194
• Add support for FP8 on compute capability >=8.0, <8.9 by @danieldk in #2213
• fix: append DONE message to chat stream by @drbh in #2221
• [fix] Modifying base in yarn embedding by @SeongBeomLEE in #2212
• Use symmetric quantization in the quantize subcommand by @danieldk in #2120
• feat: simple mistral lora integration tests by @drbh in #2180
• fix custom cache dir by @ErikKaum in #2226
• fix: Remove bitsandbytes installation when running cpu-only install by @Hugoch in #2216
• Add support for AWQ-quantized Idefics2 by @danieldk in #2233
• server quantize: expose groupsize option by @danieldk in #2225
• Remove stray quantize argument in get_weights_col_packed_qkv by @danieldk in #2237
• fix(server): fix cohere by @OlivierDehaene in #2249
• Improve the handling of quantized weights by @danieldk in #2250
• Hotfix: fix of use of unquantized weights in Gemma GQA loading by @danieldk in #2255
• Hotfix: various GPT-based model fixes by @danieldk in #2256
• Hotfix: fix MPT after recent refactor by @danieldk in #2257
• Hotfix: pass through model revision in VlmCausalLM by @danieldk in #2258
• usage stats and crash reports by @ErikKaum in #2220
• add usage stats to toctree by @ErikKaum in #2260
• fix: adjust default tool choice by @drbh in #2244
• Add support for Deepseek V2 by @danieldk in #2224
• re-push to internal registry by @XciD in #2242
• Add FP8 release test by @danieldk in #2261
• feat(fp8): use fbgemm kernels and load fp8 weights directly by @OlivierDehaene in #2248
• fix(server): fix deepseekv2 loading by @OlivierDehaene in #2266
• Hotfix: fix of use of unquantized weights in Mixtral GQA loading by @icyxp in #2269
• legacy warning on text_generation client by @ErikKaum in #2271
• fix(ci): test new instances by @XciD in #2272
• fix(server): fix fp8 weight loading by @OlivierDehaene in #2268
• Softcapping for gemma2. by @Narsil in #2273
• use proper name for ci by @XciD in #2274
• Fixing mistral nemo. by @Narsil in #2276
• fix(l4): fix fp8 logic on l4 by @OlivierDehaene in #2277
• Add support for repacking AWQ weights for GPTQ-Marlin by @danieldk in #2278
• [WIP] Add support for Mistral-Nemo by supporting head_dim through config by @shaltielshmid in #2254
• Preparing for release. by @Narsil in #2285
• Add support for Llama 3 rotary embeddings by @danieldk in #2286
• hotfix: pin numpy by @danieldk in #2289

New Contributors

• @jaluma made their first contribution in #2178
• @vinkamath made their first contribution in #2196
• @ErikKaum made their first contribution in #2226
• @Hugoch made their first contribution in #2216
• @XciD made their first contribution in #2242
• @shaltielshmid made their first contribution in #2254

Full Changelog: v2.1.1...v2.2.0

v2.1.1

Main changes

• Bugfixes
• Added FlashDecoding support (Beta) use FLASH_DECODING=1 to use TGI with flash decoding (large speedups on long queries). #1940
• Use Marlin over GPTQ kernels for faster GPTQ inference #2111

What's Changed

• Fixing the CI to also run in release when it's a tag ? by @Narsil in #2138
• fix microsoft/Phi-3-mini-4k-instruct crash in batch.slots[batch.slot_… by @sywangyi in https://github.com//pull/2148
• Fixing clippy. by @Narsil in #2149
• fix: use weights from base_layer by @drbh in #2141
• feat: download lora adapter weights from launcher by @drbh in #2140
• Use GPTQ-Marlin for supported GPTQ configurations by @danieldk in #2111
• fix AttributeError: 'MixtralLayer' object has no attribute 'mlp' by @icyxp in #2123
• refine get xpu free memory/enable Qwen2/gemma2/gemma/phi in intel platform by @sywangyi in #2132
• fix: prefer serde structs over custom functions by @drbh in #2127
• Fixing test. by @Narsil in #2152
• GH router. by @Narsil in #2153
• Fixing baichuan override. by @Narsil in #2158
• [Major Change][Undecided yet] Move to FlashDecoding instead of PagedAttention kernel. by @Narsil in #1940
• Fixing graph capture for flash decoding. by @Narsil in #2163
• fix FlashDecoding change's regression in intel platform by @sywangyi in #2161
• fix: use the base layers weight in mistral rocm by @drbh in #2155
• Fixing rocm. by @Narsil in #2164
• Ci test by @glegendre01 in #2124
• Hotfixing qwen2 and starcoder2 (which also get clamping). by @Narsil in #2167
• feat: improve update_docs for openapi schema by @drbh in #2169
• Fixing the dockerfile warnings. by @Narsil in #2173
• Fixing missing object field for regular completions. by @Narsil in #2175

New Contributors

• @icyxp made their first contribution in #2123

Full Changelog: v2.1.0...v2.1.1

v2.1.0

Notable changes

• New models : gemma2

• Multi lora adapters. You can now run multiple loras on the same TGI deployment #2010

• Faster GPTQ inference and Marlin support (up to 2x speedup).

• Reworked the entire scheduling logic (better block allocations, and allowing further speedups in new releases)

• Lots of Rocm support and bugfixes,

• Lots of new contributors ! Thanks a lot for these contributions

What's Changed

• OpenAI function calling compatible support by @phangiabao98 in #1888
• Fixing types. by @Narsil in #1906
• Types. by @Narsil in #1909
• Fixing signals. by @Narsil in #1910
• Removing some unused code. by @Narsil in #1915
• MI300 compatibility by @fxmarty in #1764
• Add TGI monitoring guide through Grafana and Prometheus by @fxmarty in #1908
• Update grafana template by @fxmarty in #1918
• Fix TunableOp bug by @fxmarty in #1920
• Fix TGI issues with ROCm by @fxmarty in #1921
• Fixing the download strategy for ibm-fms by @Narsil in #1917
• ROCm: make CK FA2 default instead of Triton by @fxmarty in #1924
• docs: Fix grafana dashboard url by @edwardzjl in #1925
• feat: include token in client test like server tests by @drbh in #1932
• Creating doc automatically for supported models. by @Narsil in #1929
• fix: use path inside of speculator config by @drbh in #1935
• feat: add train medusa head tutorial by @drbh in #1934
• reenable xpu for tgi by @sywangyi in #1939
• Fixing some legacy behavior (big swapout of serverless on legacy stuff). by @Narsil in #1937
• Add completion route to client and add stop parameter where it's missing by @thomas-schillaci in #1869
• Improving the logging system. by @Narsil in #1938
• Fixing codellama loads by using purely AutoTokenizer. by @Narsil in #1947
• Fix seeded output. by @Narsil in #1949
• Fix (flash) Gemma prefix and enable tests by @danieldk in #1950
• Fix GPTQ for models which do not have float16 at the default dtype (simpler) by @danieldk in #1953
• Processor config chat template by @drbh in #1954
• fix small typo and broken link by @MoritzLaurer in #1958
• Upgrade to Axum 0.7 and Hyper 1.0 (Breaking change: disabled ngrok tunneling). by @Narsil in #1959
• Fix (non-container) pytest stdout buffering-related lock-up by @danieldk in #1963
• Fixing the text part from tokenizer endpoint. by @Narsil in #1967
• feat: adjust attn weight loading logic by @drbh in #1975
• Add support for exl2-quantized models by @danieldk in #1965
• Update documentation version to 2.0.4 by @fxmarty in #1980
• Purely refactors paged/attention into layers/attention and make hardware differences more obvious with 1 file per hardware. by @Narsil in #1986
• Fixing exl2 scratch buffer. by @Narsil in #1990
• single char ` addition for docs by @nbroad1881 in #1989
• Fixing GPTQ imports. by @Narsil in #1994
• reable xpu, broken by gptq and setuptool upgrade by @sywangyi in #1988
• router: send the input as chunks to the backend by @danieldk in #1981
• Fix Phi-2 with tp>1 by @danieldk in #2003
• fix: update triton implementation reference by @emmanuel-ferdman in #2002
• feat: add SchedulerV3 by @OlivierDehaene in #1996
• Support GPTQ models with column-packed up/gate tensor by @danieldk in #2006
• Making make install work better by default. by @Narsil in #2004
• Hotfixing make install. by @Narsil in #2008
• Do not initialize scratch space when there are no ExLlamaV2 layers by @danieldk in #2015
• feat: move allocation logic to rust by @OlivierDehaene in #1835
• Fixing rocm. by @Narsil in #2021
• Fix GPTQWeight import by @danieldk in #2020
• Update version on init.py to 0.7.0 by @andimarafioti in #2017
• Add support for Marlin-quantized models by @danieldk in #2014
• marlin: support tp>1 when group_size==-1 by @danieldk in #2032
• marlin: improve build by @danieldk in #2031
• Internal runner ? by @Narsil in #2023
• Xpu gqa by @sywangyi in #2013
• server: use chunked inputs by @danieldk in #1985
• ROCm and sliding windows fixes by @fxmarty in #2033
• Add Phi-3 medium support by @danieldk in #2039
• feat(ci): add trufflehog secrets detection by @McPatate in #2038
• fix(ci): remove unnecessary permissions by @McPatate in #2045
• Update LLMM1 bound by @fxmarty in #2050
• Support chat response format by @drbh in #2046
• fix(server): fix OPT implementation by @OlivierDehaene in #2061
• fix(layers): fix SuRotaryEmbedding by @OlivierDehaene in #2060
• PR #2049 CI run by @drbh in #2054
• implement Open Inference Protocol endpoints by @drbh in #1942
• Add support for GPTQ Marlin by @danieldk in #2052
• Update the link for qwen2 by @xianbaoqian in #2068
• Adding architecture document by @tengomucho in #2044
• Support different image sizes in prefill in VLMs by @danieldk in #2065
• Contributing guide & Code of Conduct by @LysandreJik in #2074
• fix build.rs watch files by @zirconium-n in #2072
• Set maximum grpc message receive size to 2GiB by @danieldk in #2075
• CI: Tailscale improvements by @glegendre01 in #2079
• CI: pass pre-commit hooks again by @danieldk in #2084
• feat: rotate tests ci token by @drbh in #2091
• Support exl2-quantized Qwen2 models by @danieldk in #2085
• Factor out sharding of packed tensors by @...

v2.0.4

Main changes

• AMD MI300 compatibility by @fxmarty in #1764
• Many bugfixes.

What's Changed

• OpenAI function calling compatible support by @phangiabao98 in #1888
• Fixing types. by @Narsil in #1906
• Types. by @Narsil in #1909
• Fixing signals. by @Narsil in #1910
• Removing some unused code. by @Narsil in #1915
• MI300 compatibility by @fxmarty in #1764
• Add TGI monitoring guide through Grafana and Prometheus by @fxmarty in #1908
• Update grafana template by @fxmarty in #1918
• Fix TunableOp bug by @fxmarty in #1920
• Fix TGI issues with ROCm by @fxmarty in #1921
• Fixing the download strategy for ibm-fms by @Narsil in #1917
• ROCm: make CK FA2 default instead of Triton by @fxmarty in #1924
• docs: Fix grafana dashboard url by @edwardzjl in #1925
• feat: include token in client test like server tests by @drbh in #1932
• Creating doc automatically for supported models. by @Narsil in #1929
• fix: use path inside of speculator config by @drbh in #1935
• feat: add train medusa head tutorial by @drbh in #1934
• reenable xpu for tgi by @sywangyi in #1939
• Fixing some legacy behavior (big swapout of serverless on legacy stuff). by @Narsil in #1937
• Add completion route to client and add stop parameter where it's missing by @thomas-schillaci in #1869
• Improving the logging system. by @Narsil in #1938
• Fixing codellama loads by using purely AutoTokenizer. by @Narsil in #1947

New Contributors

• @phangiabao98 made their first contribution in #1888
• @edwardzjl made their first contribution in #1925
• @thomas-schillaci made their first contribution in #1869

Full Changelog: v2.0.3...v2.0.4

v2.0.3

Important changes

• Add: Support for the Falcon2 by @Nilabhra in #1886
• New speculation method MLPSpeculator. by @JRosenkranz in #1865
• Pali gemma modeling by @drbh in #1895

What's Changed

• Fix: "Fixing" double BOS for mistral too. by @Narsil in #1843
• Adding scripts to prepare load data. by @Narsil in #1841
• Remove misleading warning (not that important nowadays anyway). by @Narsil in #1848
• feat: prefer huggingface_hub in docs and show image api by @drbh in #1844
• Updating Phi3 (long context). by @Narsil in #1849
• Add router name to /info endpoint by @Wauplin in #1854
• Upgrading to rust 1.78. by @Narsil in #1851
• update xpu docker image and use public ipex whel by @sywangyi in #1860
• Refactor layers. by @Narsil in #1866
• Granite support? by @Narsil in #1882
• Add: Support for the Falcon2 11B architecture by @Nilabhra in #1886
• MLPSpeculator. by @JRosenkranz in #1865
• Fixing truncation. by @Narsil in #1890
• Correct 'using guidance' link by @brandon-lockaby in #1892
• Add GPT-2 with flash attention by @danieldk in #1889
• Removing accepted ids in the regular info logs, downgrade to debug. by @Narsil in #1898
• feat: add deprecation warning to clients by @drbh in #1855
• [Bug Fix] Update torch import reference in bnb quantization by @DhruvSrikanth in #1902
• Pali gemma modeling by @drbh in #1895

New Contributors

• @Nilabhra made their first contribution in #1886
• @brandon-lockaby made their first contribution in #1892
• @danieldk made their first contribution in #1889
• @DhruvSrikanth made their first contribution in #1902

Full Changelog: v2.0.2...v2.0.3

v2.0.2

Tl;dr

• New models (idefics2, phi3)
• Cleaner VLM support in the openai layer
• Upgraded to pytorch 2.3.0

What's Changed

• Make --cuda-graphs 0 work as expected (bis) by @fxmarty in #1768
• fix typos in docs and add small clarifications by @MoritzLaurer in #1790
• Add attribute descriptions for GenerateParameters by @Wauplin in #1798
• feat: allow null eos and bos tokens in config by @drbh in #1791
• Phi3 support by @Narsil in #1797
• Idefics2. by @Narsil in #1756
• fix: avoid frequency and repetition penalty on padding tokens by @drbh in #1765
• Adding support for HF_HUB_OFFLINE support in the router. by @Narsil in #1789
• feat: improve temperature logic in chat by @drbh in #1749
• Updating the benchmarks so everyone uses openai compat layer. by @Narsil in #1800
• Update guidance docs to reflect grammar support in API by @dr3s in #1775
• Use the generation config. by @Narsil in #1808
• 2nd round of benchmark modifications (tiny adjustements to avoid overloading the host). by @Narsil in #1816
• Adding new env variables for TPU backends. by @Narsil in #1755
• add intel xpu support for TGI by @sywangyi in #1475
• Blunder by @Narsil in #1815
• Fixing qwen2. by @Narsil in #1818
• Dummy CI run. by @Narsil in #1817
• Changing the waiting_served_ratio default (stack more aggressively by default). by @Narsil in #1820
• Better graceful shutdown. by @Narsil in #1827
• Add the missing tool_prompt parameter to Python client by @maziyarpanahi in #1825
• Small CI cleanup. by @Narsil in #1801
• Add reference to TPU support by @brandonroyal in #1760
• fix: use get_speculate to the number of layers by @OlivierDehaene in #1737
• feat: add how it works section by @drbh in #1773
• Fixing frequency penalty by @martinigoyanes in #1811
• feat: add vlm docs and simple examples by @drbh in #1812
• Handle images in chat api by @drbh in #1828
• chore: update torch by @OlivierDehaene in #1730
• (chore): torch 2.3.0 by @Narsil in #1833

New Contributors

• @MoritzLaurer made their first contribution in #1790
• @dr3s made their first contribution in #1775
• @maziyarpanahi made their first contribution in #1825
• @brandonroyal made their first contribution in #1760
• @martinigoyanes made their first contribution in #1811

Full Changelog: v2.0.1...v2.0.2

v2.0.1

What's Changed

• feat: improve tools to include name and add tests by @drbh in #1693
• Update response type for /v1/chat/completions and /v1/completions by @Wauplin in #1747
• accept list as prompt for OpenAI API by @drbh in #1702
• fix ROCm docker image

Full Changelog: v2.0.0...v2.0.1

v2.0.0

TGI is back to Apache 2.0!

Highlights

• License was reverted to Apache 2.0
• Cuda graphs are now used by default. They improve latency substancially on high end nodes.
• Llava-next was added. It is the second multimodal model available on TGI after Idefics.
• Cohere Command R+ support. TGI is the fastest open source backend for Command R+
• FP8 support.
• We now share the vocabulary for all medusa heads, greatly improving latency and memory use.

Try out Command R+ with Medusa heads on 4xA100s with:

model=text-generation-inference/commandrplus-medusa
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 3 --num-shard 4

What's Changed

• Add cuda graphs sizes and make it default. by @Narsil in #1703
• Pickle conversion now requires --trust-remote-code. by @Narsil in #1704
• Push users to streaming in the readme. by @Narsil in #1698
• Fixing cohere tokenizer. by @Narsil in #1697
• Force weights_only (before fully breaking pickle files anyway). by @Narsil in #1710
• Regenerate ld.so.cache by @oOraph in #1708
• Revert license to Apache 2.0 by @OlivierDehaene in #1714
• Automatic quantization config. by @Narsil in #1719
• Adding Llava-Next (Llava 1.6) with full support. by @Narsil in #1709
• fix: fix CohereForAI/c4ai-command-r-plus by @OlivierDehaene in #1707
• Update libraries by @abhishekkrthakur in #1713
• Dev/mask ldconfig output v2 by @oOraph in #1716
• Fp8 Support by @Narsil in #1726
• Upgrade EETQ (Fixes the cuda graphs). by @Narsil in #1729
• fix(router): fix a possible deadlock in next_batch by @OlivierDehaene in #1731
• chore(cargo-toml): apply lto fat and codegen-units of one by @somehowchris in #1651
• Improve the defaults for the launcher by @Narsil in #1727
• feat: medusa shared by @OlivierDehaene in #1734
• Fix typo in guidance.md by @eltociear in #1735

New Contributors

• @somehowchris made their first contribution in #1651

Full Changelog: v1.4.5...v2.0.0

v.1.4.5

Highlights

• DBRX support #1685. See #1679 on how to prompt the model.

What's Changed

• fix: adjust logprob response logic by @drbh in #1682
• fix: handle batches with and without grammars by @drbh in #1676
• feat: Add dbrx support by @OlivierDehaene in #1685

Full Changelog: v1.4.4...v1.4.5

v.1.4.4

Highlights

• CohereForAI/c4ai-command-r-v01 model support

What's Changed

• Handle concurrent grammar requests by @drbh in #1610
• Fix idefics default. by @Narsil in #1614
• Fix async client timeout by @hugoabonizio in #1617
• accept legacy request format and response by @drbh in #1527
• add missing stop parameter for chat request by @drbh in #1619
• correctly index into mask when applying grammar by @drbh in #1618
• Use a better model for the quick tour by @lewtun in #1639
• Upgrade nix version from 0.27.1 to 0.28.0 by @yuanwu2017 in #1638
• Update peft + transformers + accelerate + bnb + safetensors by @abhishekkrthakur in #1646
• Fix index in ChatCompletionChunk by @Wauplin in #1648
• Fixing minor typo in documentation: supported hardware section by @SachinVarghese in #1632
• bump minijina and add test for core templates by @drbh in #1626
• support force downcast after FastRMSNorm multiply for Gemma by @drbh in #1658
• prefer spaces url over temp url by @drbh in #1662
• improve tool type, bump pydantic and outlines by @drbh in #1650
• Remove unecessary cuda graph. by @Narsil in #1664
• Repair idefics integration tests. by @Narsil in #1663
• fix: LlamaTokenizerFast to AutoTokenizer at flash_mistral.py by @SeongBeomLEE in #1637
• Inline images for multimodal models. by @Narsil in #1666

New Contributors

• @hugoabonizio made their first contribution in #1617
• @yuanwu2017 made their first contribution in #1638
• @abhishekkrthakur made their first contribution in #1646
• @Wauplin made their first contribution in #1648
• @SachinVarghese made their first contribution in #1632
• @SeongBeomLEE made their first contribution in #1637

Full Changelog: v1.4.3...v1.4.4