Add Integer sequence generator

[rlouf]

In this PR I introduce a sequence generator that only outputs integers. Closes #153.

The generator can currently generate sequences that have leading zeros, which are then trimmed during post-processing. I am not sure what we should do about it. On one hand we can ignore it (maybe the LLM has seen this during training), on the other hand we could use the pre-processing approach described in #131.

[rlouf]

The code now makes use of the tools introduced in #131 and the follow-up #170 to pre-compute the masks and advance the parsing state. We allow the EOS to be generated as well, and forbid leading zeros.

TODO

• Move torch.nn.softmax from Transformers.__call__ to Sequence.step after creating the proposal
• Make this work for batch generation

[brandonwillard]

Minor comments/questions; looks good otherwise.

[brandonwillard]

Just a reminder that we can avoid reparsing from the beginning of the sequence, but that's fine as a follow-up.

[rlouf]

The integer generation is working, although we currently cannot handle the end-of-sequence (EOS) token properly. To generate integers, we'd like the generated text to match the regex (0|[1-9][0-9]+)(<EOS>) where <EOS> is the EOS token. The corresponding finite-state machine contains the transitions "<" -> "E" -> "O" -> "S" -> ">", but we can be in a situation where for instance <EOS is in the vocabulary but > is not, stalling the generation. Even if > were part of the vocabulary the generation would still be wrong as we only want to allow transitions to the token id that corresponds to <EOS>.

Since EOS is always a final state we shouldn't have too much trouble modifying the alphabet/map of the FSM to get the desired behavior. This is non blocking, will open an issue that will need a follow-up PR.