[BT] add decoder benchmark script
#857
Conversation
| max_new_tokens=max_token, | ||
| use_cache=False, |
There was a problem hiding this comment.
Why use_cache = False? I'd rather benchmark with the cache enabled.
Also, can we rather set min_length=length, max_length=length? So that we can generate a specific length.
There was a problem hiding this comment.
I think it's because max_length is deprecated, yes we can add min_length I think let me try
| max_new_tokens=max_token, | ||
| use_cache=False, | ||
| ) | ||
| _ = hf_model.generate(input_ids, generation_config=gen_config) |
There was a problem hiding this comment.
the input_ids are not defined no?
There was a problem hiding this comment.
Does it make sense to pass attention_mask as well? Not sure
| start_event = torch.cuda.Event(enable_timing=True) | ||
| end_event = torch.cuda.Event(enable_timing=True) | ||
| start_event.record() | ||
| for _ in range(num_batches): | ||
| _ = model(input_ids, masks) | ||
| if is_decoder: | ||
| _ = model.generate(input_ids, generation_config=generation_config) |
There was a problem hiding this comment.
the input_ids is not defined no?
|
The documentation is not available anymore as the PR was closed or merged. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
What does this PR do?
This PR adds the benchmark script adapted for decoder-based models as well, to benchmark the speedup we obtain with
torch.sdpaandtransformersmodelscc @fxmarty