Fix Incorrect Token Normalization Method for LlamaCppTokenizer

[lapp0]

Fixes #952

Problem

During FSM index creation, the normalized vocabulary is used to determine token validity for each state in the automata.

Because the set of token strings in tokenizer.vocabulary isn't equivalent to the token strings created by tokenizer.decode the vocabulary tokens must be normalized to ensure the FSM selects tokens from the vocabulary which, when decoded, conform to the pattern.

In models.llamacpp there wasn't any normalization, and the \n token is represented in vocabulary as Ċ

>>> tokenizer.vocabulary['Ċ']
198
>>> tokenizer.decode([198])
['\n']

During FSM compilation, Outlines understood "Ċ" to be a valid json string, however in reality, the invalid json string "\n" was decoded.

Solution

Implement working normalization function, convert_token_to_string(), to ensure behavior is correct when using a huggingface transformers tokenizer with models.llamacpp.