Convert tokens to strings before partial matching

[rlouf]

BPE tokenizers encode whitespaces as a special character Ġ, and it is possible that another tokenizer may encode one or several strings as special character(s). Partial matching with the tokens directly thus prevents relevant matches from happening.

We thus add a convert_token_to_string method that converts back these special character to the string they correspond to so we can partially match the corresponding tokens with the regex.

The following snippet illustrates the problem and its solution:

import interegular

from outlines.text.parsing import find_partial_matches
from outlines.models.transformers import transformers

model = transformers("gpt2")

fsm = interegular.parse_pattern("This is a new day").to_fsm()
pmatch = find_partial_matches(fsm, "Ġday")
print(pmatch)
# set()

pmatch = find_partial_matches(fsm, model.tokenizer.convert_token_to_string("Ġday"))
print(pmatch)
# {(3, (13, 14, 15, 16, 17))}

Note: we could also iterate over the token ids and use tokenizer.decode. This may be simpler than adding a new function, especially since we wouldn't have to sort the vocabulary. But maybe less explicit as this may leave people wondering why we don't use vocabulary.keys() directly.