-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate a choice between different strings #188
Conversation
9228675
to
c1d7a61
Compare
for i, vocab_string in enumerate(vocabulary): | ||
if vocab_string == final_state_string: | ||
final_state_string_idx = i | ||
|
||
for end_idx, state_seq in find_partial_matches(fsm, vocab_string): | ||
if partial_match_filter(vocab_string, end_idx, state_seq): | ||
terminal_possible_paths[state_seq[0]].add(state_seq[-1]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whether or not the last state is a final/accept state of the FSM will depend on partial_match_filter
, no? We might need extra handling for this, or we could track these things in that filter function, after adding the symbol name as an argument, of course.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The check would fail if the combination of the vocabulary and partial_match_filter
didn't allow to reach a final state, no? Isn't that the desired behaviour?
a36c5c2
to
b29a601
Compare
If the vocabulary does not allow to transition from the FSM's initial state to one of its terminal paths it will be impossible to generate a sequence that matches the input regex. We therefore raise an exception when no such transition exists.
b29a601
to
a3f7b9e
Compare
Closes #152. I use the
Regex
generation method using a(choice_1|choice_2|...|choice_n)
regex. This method is equivalent to rejection sampling: draw a token and discard it if it is not on a path that generates either of the choices.An alternative implementation would be to tokenize the choices, do a forward pass through the model with each and sample from the resulting categorical distribution. However, some strings can be built from different combinations of tokens and we would only be considering one for each stopping sequence.
Unsurprisingly, the check to verify that a sequence can be generated with the vocabulary added in #175 fails. We fix this by checking whether we can build terminal paths with the vocabulary. Closes #184.