Fix missing spaces in Tokenizer.convert_token_to_string

[brandonwillard]

Closes #273

[brandonwillard]

This probably isn't a general enough fix, but it should be a workable stand-in for Llama models right now, so I'm going to merge this. @AL-377, still feel free to put in a PR for another approach, especially if you think it will cover more cases.

[AL-377]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

[brandonwillard]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

Which HF tokenizer class is it using?

[AL-377]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

Which HF tokenizer class is it using?

transformers.models.code_llama.tokenization_code_llama_fast.CodeLlamaTokenizerFast

[brandonwillard]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

Which HF tokenizer class is it using?

transformers.models.code_llama.tokenization_code_llama_fast.CodeLlamaTokenizerFast

Thanks! We just need to add CodeLlamaTokenizerFast to the isinstance check in this PR.

[AL-377]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

Which HF tokenizer class is it using?

transformers.models.code_llama.tokenization_code_llama_fast.CodeLlamaTokenizerFast

Thanks! We just need to add CodeLlamaTokenizerFast to the isinstance check in this PR.

But actually I have tried the code:

self.is_sentencepiece = isinstance(
            self.tokenizer, (LlamaTokenizerFast, LlamaTokenizer,CodeLlamaTokenizer,CodeLlamaTokenizerFast)
        )

    And I got the error below:

 │ /outlines/text/generate/regex.py:118 in create_proposal                  │
│                                                                                                  │
│   115 │   │   │   │   │                                                                          │
│   116 │   │   │   │   │   sequence = self.model.tokenizer.decode(readable_tokens)                │
│   117 │   │   │   │   │                                                                          │
│ ❱ 118 │   │   │   │   │   ((_, state_seq),) = find_partial_matches(                              │
│   119 │   │   │   │   │   │   self.regex_fsm,                                                    │
│   120 │   │   │   │   │   │   "".join(sequence),                                                 │
│   121 │   │   │   │   │   │   start_state=last_fsm_state,                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: not enough values to unpack (expected 1, got 0)

My test.py is just the same as the example at https://github.com/outlines-dev/outlines#efficient-json-generation-following-a-pydantic-model

[brandonwillard]

OK,thanks for your effort！I have tried the codellama model and the result remains the white space,I will keep trying

Which HF tokenizer class is it using?

transformers.models.code_llama.tokenization_code_llama_fast.CodeLlamaTokenizerFast

Thanks! We just need to add CodeLlamaTokenizerFast to the isinstance check in this PR.

But actually I have tried the code:

self.is_sentencepiece = isinstance(
            self.tokenizer, (LlamaTokenizerFast, LlamaTokenizer,CodeLlamaTokenizer,CodeLlamaTokenizerFast)
        )

    And I got the error below:

 │ /outlines/text/generate/regex.py:118 in create_proposal                  │
│                                                                                                  │
│   115 │   │   │   │   │                                                                          │
│   116 │   │   │   │   │   sequence = self.model.tokenizer.decode(readable_tokens)                │
│   117 │   │   │   │   │                                                                          │
│ ❱ 118 │   │   │   │   │   ((_, state_seq),) = find_partial_matches(                              │
│   119 │   │   │   │   │   │   self.regex_fsm,                                                    │
│   120 │   │   │   │   │   │   "".join(sequence),                                                 │
│   121 │   │   │   │   │   │   start_state=last_fsm_state,                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: not enough values to unpack (expected 1, got 0)

My test.py is just the same as the example at https://github.com/outlines-dev/outlines#efficient-json-generation-following-a-pydantic-model

Yeah, that looks like a different issue with the Llama models altogether. I'll check it out.

[brandonwillard]

That issue is due to tokens that correspond to empty strings (see #273 (comment)). The general issue of empty strings and our regex sampling logic might go beyond this Llama-specific tokenization issue, though. I address it directly in #272, which replaces and simplifies all the regex logic.