New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TextGeneration] Fix internal kv_cache update for batch_size > 1 #1514

Merged

bfineran merged 1 commit into main from fix_internal_kv_cache_update

Jan 5, 2024

Contributor

dsikka commented Jan 5, 2024 •

edited

Loading

Summary

When updating the NLEngineOperator to process batch_size > 1 with internal_kv_cache enabled (for continuous_batching), one line was missed in the update PR. This PR fixes this issue and also cleans up some of the conditions
This will also fix this ticket:
https://app.asana.com/0/1205229323407165/1206239324632274/f


          internal kv_cache update for batch_size > 1

b640808

bfineran approved these changes

View reviewed changes

bfineran merged commit 05a47b0 into main

13 checks passed

bfineran deleted the fix_internal_kv_cache_update branch

January 5, 2024 22:42

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment