TensorRT-LLM Engine integration #3228
Conversation
| maxBatchDelay: 100 | ||
| responseTimeout: 1200 | ||
| deviceType: "gpu" | ||
| asyncCommunication: true |
There was a problem hiding this comment.
Can TRT-LLM handle multi-gpu inference easily? Then we should demonstrate that we can easily integrate that with
parallelType: "custom"
parallelLevel: 4
There was a problem hiding this comment.
I tried standalone multi gpu inference, it didn't work for me. Although the model loaded on the 4 GPUs, the inference was hanging
| streaming=True, | ||
| return_dict=True, | ||
| ) | ||
| torch.cuda.synchronize() |
There was a problem hiding this comment.
What is the synchronization for?
There was a problem hiding this comment.
Copy paste from the example code , but seems like its not needed. Works fine without it
| for beam in range(num_beams): | ||
| output_begin = input_lengths[batch_idx] | ||
| output_end = sequence_lengths[batch_idx][beam] | ||
| outputs = output_ids[batch_idx][beam][ |
There was a problem hiding this comment.
Are we sure we're doing the right thing here? Because output_begin is never used but instead output_end-1.
There was a problem hiding this comment.
good catch, we don't need output_begin. The example code uses this
| output_end - 1 : output_end | ||
| ].tolist() | ||
| output_text = self.tokenizer.decode(outputs) | ||
| send_intermediate_predict_response( |
There was a problem hiding this comment.
I we send N=num_beams intermediate results back without any order information, can we assign each partial response to its beam sequence? Better to send one response per batch entry (will will be =1) with updates for all beams included as a list?
There was a problem hiding this comment.
it seems like this is not needed for llama as num_beams > 1 is not working for llama Removed the inner for loop
| @@ -10,6 +10,7 @@ This will downgrade the versions of PyTorch & Triton but this doesn't cause any | |||
|
|
|||
| ``` | |||
| pip install tensorrt_llm==0.10.0 --extra-index-url https://pypi.nvidia.com | |||
| pip install tensorrt-cu12==10.1.0 | |||
There was a problem hiding this comment.
Is this CUDA 12 exclusive? In that case we should inform people to install torch with CUDA 12 as well.
There was a problem hiding this comment.
It doesn't mention it but all their docs are pointing to CUDA 12.x . Let me mention that its tested with CUDA 12.1
Description
This PR shows how to integrate TensorRT-LLM Engine with TorchServe
Fixes #(issue)
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Test A
Logs for Test A
Test B
Logs for Test B
Checklist: