-
Notifications
You must be signed in to change notification settings - Fork 412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix evaluation code to improve performance #2421
Comments
Hi, @vigneshwaran, thanks for discovering this! please make a pull request with your changes! |
Great find! If you can open a PR, we'd love to accept it :) |
This was referenced Aug 10, 2023
Closed
I have proposed my MR #2428 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have been experiencing llm-foundry/eval takes a lot of time compared to lm-evaluation-harness. After digging into the code, I found padding token is appended till the maximum length of the tokenizer.
https://github.com/bmosaicml/composer/blob/1011f90f2653dae103c3837c968071e399b1decc/composer/datasets/in_context_learning_evaluation.py#L418C1-L428C59
My proposal:
Instead of padding till max_seq_len, use the maximum length of the batch.
This has improved latency by 400% when I used 2048 as sequence length. It would be even more for models trained with higher sequence length.
The text was updated successfully, but these errors were encountered: