-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Integer
sequence generator
#166
Conversation
88ebc06
to
47b236a
Compare
1560beb
to
9cdc0a0
Compare
The code now makes use of the tools introduced in #131 and the follow-up #170 to pre-compute the masks and advance the parsing state. We allow the TODO
|
1705594
to
13ad481
Compare
7e7b635
to
130ec8b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor comments/questions; looks good otherwise.
outlines/text/generate/integer.py
Outdated
""" | ||
if generated_token_ids.shape[-1] > 0: | ||
sampled_sequence = self.model.tokenizer.decode(generated_token_ids) | ||
partial_matches = find_partial_matches(self.int_regex_fsm, sampled_sequence) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a reminder that we can avoid reparsing from the beginning of the sequence, but that's fine as a follow-up.
f43cc43
to
f1cb953
Compare
f1cb953
to
fd98f26
Compare
The integer generation is working, although we currently cannot handle the end-of-sequence (EOS) token properly. To generate integers, we'd like the generated text to match the regex Since EOS is always a final state we shouldn't have too much trouble modifying the alphabet/map of the FSM to get the desired behavior. This is non blocking, will open an issue that will need a follow-up PR. |
In this PR I introduce a sequence generator that only outputs integers. Closes #153.
The generator can currently generate sequences that have leading zeros, which are then trimmed during post-processing. I am not sure what we should do about it. On one hand we can ignore it (maybe the LLM has seen this during training), on the other hand we could use the pre-processing approach described in #131.