Dolphin Mistral Prompt blows up on assertion

[chris-cortner]

I'm trying to apply dolphin mistral's prompt template format:

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant

I've tried this a couple of different ways:

quant_path = "TheBloke/dolphin-2.6-mistral-7B-AWQ"
lm = models.Transformers(quant_path, device_map="auto")
stop_char = '"'
prompt_template = '<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n'

lm2 = lm + (prompt_template.format(system_prompt="You are a helpful AI", prompt="What is the distance to mars?")
+ f'The distance to mars is "{gen("answer", max_tokens=500, stop=stop_char, temperature=0.7)}"')

And by using TransformersChat:

quant_path = "TheBloke/dolphin-2.6-mistral-7B-AWQ"
lm = models.TransformersChat(quant_path, device_map="auto")
stop_char = '"'

with system():
lm2 = lm + "You are a helpful AI"

with user():
lm2 += "What is the distance to mars?"

with assistant():
lm2 += 'The distance to mars is "' + gen("answer", max_tokens=500, stop=stop_char, temperature=0.8)

Both method produce the same error:

An assertion error is thrown in _cleanup_tokens in _model.py

Traceback (most recent call last):
File "/home/user/.cache/pypoetry/virtualenvs/llm-proficiency-testing-hKJXaDzo-py3.11/lib64/python3.11/site-packages/guidance/models/_model.py", line 309, in add
out = lm + partial_grammar
~~~^~~~~~~~~~~~~~~~~
File "/home/user/.cache/pypoetry/virtualenvs/llm-proficiency-testing-hKJXaDzo-py3.11/lib64/python3.11/site-packages/guidance/models/_model.py", line 317, in add
out = lm._run_stateless(value)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.cache/pypoetry/virtualenvs/llm-proficiency-testing-hKJXaDzo-py3.11/lib64/python3.11/site-packages/guidance/models/_model.py", line 482, in _run_stateless
for new_bytes, is_generated, new_bytes_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
File "/home/user/.cache/pypoetry/virtualenvs/llm-proficiency-testing-hKJXaDzo-py3.11/lib64/python3.11/site-packages/guidance/models/_model.py", line 798, in call
token_ids,token_byte_positions = self._cleanup_tokens(token_ids, token_byte_positions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.cache/pypoetry/virtualenvs/llm-proficiency-testing-hKJXaDzo-py3.11/lib64/python3.11/site-packages/guidance/models/_model.py", line 628, in _cleanup_tokens
assert token_byte_positions[-1] == last_pos
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Please let me know if I should provide more information.

[chris-cortner]

Looks like a related issue was previously closed but another person was still reporting the issue in v0.1.10
Issue #556

Also, I'm running from main as of 2 days ago.

[chris-cortner]

Seems to work fine with neural-chat, which is Mistral based but uses a different system prompt format:

"### System:\n{system_prompt}\n\n### User:\n{prompt}\n\n### Assistant:\n"

[aalyousfi]

I'm facing a similar issue, I get an assertion error when using the ChatML format. Is there a fix for that?

[chris-cortner]

I also added a discussion topic, hoping someone will chime in with a workaround.

[RXminuS]

I'm facing a similar issue but it works if I use LlamaCppChat model instead of HuggingFaceChat model. Also it's specifically to do with the<| and |> because if I replace them with << and >> the prompt works just fine.

[chris-cortner]

Very interesting, thanks for the tip!