huggingface text-generation-inference · Discussions · GitHub

Is there a way to get GenerateResponse full json when calling sagemaker endpoints
Q&A nth-attempt

Categories

Discussions

You must be logged in to vote

Do we have any Grafana Dashboard example on top of TGI metrics ? Could metrics be explained in more detail ?

luk-cer started Oct 10, 2023 in Ideas · Closed

14
You must be logged in to vote

Is TGI not available for ARM64 platform?

SMAntony asked Jul 18, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Loading LORA Adapters with specific configuration

bely66 asked Jul 15, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Tagging a request and the corresponding response

mehul-ion asked Jul 11, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Loading Llama-3-70B-Instruct-AWQ with larger context across two GPUs

daytonturner asked Jun 19, 2024 in Q&A · Unanswered

0
You must be logged in to vote

text-generation-benchmark support accuracy benchmark

yiliu30 asked Jun 11, 2024 in Q&A · Unanswered

0
You must be logged in to vote

How to control request size within a batch?

cringelord000222 asked Jun 6, 2024 in Q&A · Unanswered

0
You must be logged in to vote

OTLP Endpoint Setup

devidw asked Nov 8, 2023 in Q&A · Unanswered

2
You must be logged in to vote

H100 & TGI getting 25 tokens per second?

daz-williams started May 25, 2024 in General

1
You must be logged in to vote

microsoft/Phi-3-vision-128k-instruct support

0Falli0 started May 24, 2024 in Ideas

0
You must be logged in to vote

Support for more text generation parameters

rantav asked May 19, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Idefics2: how to send a post request with uploaded image?

shizhi2 asked May 16, 2024 in Q&A · Unanswered

2
You must be logged in to vote

Question about KV cache

martinigoyanes asked May 15, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Why is my batch size maxing out at 20 when my estimated max is around 60?

ColtonBehannon asked Dec 11, 2023 in Q&A · Unanswered

3
You must be logged in to vote

NotImplementedError: Not implemented attn grouped_query_attention

zhao-lun asked May 10, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Clarification on JSON_OUTPUT field in text-generation-launcher command.

ansSanthoshM asked Feb 22, 2024 in Q&A · Unanswered

1
You must be logged in to vote

How do I use my fine-tuned version of Llava with TGI?

Iven2132 asked Apr 23, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Error for exllamav2_kernel, running TGI on Google Colab

andychoi98 asked Apr 18, 2024 in Q&A · Unanswered

4
You must be logged in to vote

Timeline Support for Volta Architecture(Tesla V100)

koffisam started Mar 27, 2024 in General

0
You must be logged in to vote

Will TGI Integrate Quanto?

PenutChen started Mar 20, 2024 in General

0
You must be logged in to vote

Do you have any plan to expand the scope to CPU?

mengniwang95 started Oct 8, 2023 in General

1
You must be logged in to vote

Helm Chart available

kjoth asked Feb 28, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Multi-GPU Inference

daz-williams started Jan 23, 2024 in General

3
You must be logged in to vote

Does HF TGI support Multi Node -Multi GPU server set up ?

ansSanthoshM asked Feb 14, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Securing Specific Routes with Token Authentication in TGI on Kubernetes

ahadrez asked Feb 10, 2024 in Q&A · Closed · Answered

1