Add outlines.generate.fsm() API entrypoint

[miftahmoha]

This adds the outlines.generate.fsm() API entrypoint following #670.

Pre-commit:

Tests:

[lapp0]

Could you please review the test cases tests/fsm/test_fsm.py and introduce test cases for this new method?

It's probably sufficient to simply verify that RegexFSM.from_interegular_fsm(interegular.parse_pattern(pattern)) == RegexFSM(pattern) for a few patterns.

[miftahmoha]

I added a type correction, using the cache, mypy wasn't able to check for Tuple[Tuple[str, int]] which should have been Tuple[Tuple[str, int], ...].

[rlouf]

Would you mind rebasing your branch on main (and overwrite the merge)? Also we need to document this new feature and briefly explain why it is useful.

[miftahmoha]

@rlouf Done.

According to #670 (comment), interegular.FSM offers operations that RegexFSM doesn't seem to have. For the examples, we've got #666 and #156 that we can use.

What do you think @lapp0?

[lapp0]

@miftahmoha yes, we should document those use cases. This perfectly solves those problems.

Use a pattern but disallow certain keywords.

Example: A list of strings, but none of those strings can be "pink" or "elephant"

import interegular

list_of_strings_pattern = """\["[^"\s]*"(?:,"[^"\s]*")*\]"""
pink_elephant_pattern = """.*(pink|elephant).*"""

list_of_strings_fsm = interegular.parse_pattern(list_of_strings_pattern).to_fsm()
pink_elephant_fsm = interegular.parse_pattern(pink_elephant_pattern).to_fsm()

list_of_strings_fsm.accepts('["a","pink","elephant"]')
# True

subtracted_fsm = list_of_strings_fsm - pink_elephant_fsm

subtracted_fsm.accepts('["a","pink","elephant"]')
# False
subtracted_fsm.accepts('["a","blue","donkey"]')
# True

If you're talking about implementing not_in, I think that's out of scope for this PR and should be discussed in a separate issue.

[miftahmoha]

@lapp0 @rlouf I added some documentation.

[rlouf]

Thank you so much for the valuable addition @miftahmoha !