Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the need to copy all tokens during basic generation #852

Merged
merged 1 commit into from
May 1, 2024

Conversation

brandonwillard
Copy link
Member

@brandonwillard brandonwillard commented Apr 30, 2024

This change removes the need to copy the entire list of token IDs during basic generation. (N.B. I'm not a fan of the use of None to indicate the full vocabulary, but it's simple and does the job.)

More can be done to improve this situation dramatically (e.g. return the original logits untouched, update the original logits array in-place using the disallowed token IDs in other cases, etc.), but a change like this is a good start.

@brandonwillard brandonwillard added optimization Related to performance optimizations text Linked to text generation enhancement labels Apr 30, 2024
@brandonwillard brandonwillard self-assigned this Apr 30, 2024
@rlouf
Copy link
Member

rlouf commented May 1, 2024

Not a huge fan of None either, but I don't have a better alternative to suggest. Otherwise looks good.

Note that this is a breaking change for libraries downstream.

@rlouf rlouf merged commit 164d1f0 into dottxt-ai:main May 1, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement optimization Related to performance optimizations text Linked to text generation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants