Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relax check for whether we can build a sequence that matches a regex #184

Closed
rlouf opened this issue Jul 13, 2023 · 0 comments · Fixed by #188
Closed

Relax check for whether we can build a sequence that matches a regex #184

rlouf opened this issue Jul 13, 2023 · 0 comments · Fixed by #188
Labels
enhancement text Linked to text generation

Comments

@rlouf
Copy link
Member

rlouf commented Jul 13, 2023

We can check whether a vocabulary allows to build a sequence that matches a regex when initializing Regex. The check is currently implemented as:

states_with_transition = {x[1] for x in pstate_to_vocab.keys()}
if len(self.regex_fsm.states.difference(states_with_transition)) > 0:
    raise ValueError(
        "The vocabulary does not allow us to build a sequence that matches the input regex"
    )

However this check might be too strict. While some states might be unreachable with a dictionary there could still be a path to terminal states. We should probably raise a warning when this happens, not an exception.

@rlouf rlouf added text Linked to text generation enhancement labels Jul 13, 2023
@rlouf rlouf changed the title Check for whether we can build a sequence that matches a regex with vocabulary Relax check for whether we can build a sequence that matches a regex Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement text Linked to text generation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant