fixing prompt template of chatml by removal of linebreak

[timothylimyl]

reference to discord chat:
"""
hi <@208256080092856321> , went through the code step-by-step and found that there's an extra linebreak ('\n') in the separator of ChatML so the separator will end up having two linebreaks . You can see it here: https://github.com/OpenAccess-AI-Collective/axolotl/blob/a581e9f8f66e14c22ec914ee792dd4fe073e62f6/src/axolotl/prompt_strategies/sharegpt.py#L16 and https://github.com/OpenAccess-AI-Collective/axolotl/blob/a48dbf6561cc74c275a48070f397334a2c367dd5/src/axolotl/monkeypatch/fastchat_conversation_turns.py#L117 . A typical silent killer of prompt template for those not aware but the model is most probably robust enough to still reply coherently since it's just a linebreak.
"""

[casper-hansen]

Could you provide an example with before and after using python -m axolotl.cli.preprocess your_config.yml --debug? Just curious to see the actual difference on a per-token level to verify that this is the case.

[winglian]

Could you provide an example with before and after using python -m axolotl.cli.preprocess your_config.yml --debug? Just curious to see the actual difference on a per-token level to verify that this is the case.

here's the current main, definitely something buggy

[casper-hansen]

Could you provide an example with before and after using python -m axolotl.cli.preprocess your_config.yml --debug? Just curious to see the actual difference on a per-token level to verify that this is the case.

here's the current main, definitely something buggy

Oh that's really bad. Essentially you are putting in a sample with just noise with <0x0A><0x0A>?

[timothylimyl]

Could you provide an example with before and after using python -m axolotl.cli.preprocess your_config.yml --debug? Just curious to see the actual difference on a per-token level to verify that this is the case.

here's the current main, definitely something buggy

I think the difference is that there's just an extra token 13.

@winglian seems like you made <|im_start|> and <|im_end|> to be special tokens, can you provide me some pointers on how did you do that in axolotl?

[NanoCode012]

@timothylimyl

seems like you made <|im_start|> and <|im_end|> to be special tokens, can you provide me some pointers on how did you do that in axolotl?

You can add them to

# bos/eos/pad/unk
special_tokens:

# others
tokens:

[NanoCode012]

I noticed this a while back but forgot about it. I think this is a good fix for this silent bug.

[timothylimyl]

@timothylimyl

seems like you made <|im_start|> and <|im_end|> to be special tokens, can you provide me some pointers on how did you do that in axolotl?

You can add them to

# bos/eos/pad/unk
special_tokens:

# others
tokens:

Is axolotl robust enough to deal with the extra tokens in the vocabulary? For example, <|im_start|> and <|im_end|> will need to be added to the tokenizer config while the model final layer has to be extended to accomodate for extra token.

[winglian]

Is axolotl robust enough to deal with the extra tokens in the vocabulary? For example, <|im_start|> and <|im_end|> will need to be added to the tokenizer config while the model final layer has to be extended to accomodate for extra token.

yes, it handles it correctly. Are you asking specifically about LoRA? (in which case you need to manually specify the lm_head and embed_tokens layers as lora_modules_to_save

[timothylimyl]

@winglian

just notice this, for the last few tokens (ending):

<|im_end|>(32000, 32000) (28705, 28705) <0x0A>(13, 13) <|im_end|>(32000, 32000)

why is there a token id 28705 there? Any clues?

[noobmaster29]

The following change seems to fix the double EOS token:

register_conv_template(
Conversation(
name="chatml",
system_template="<|im_start|>system\n{system_message}",
system_message="You are a helpful assistant.",
roles=["<|im_start|>user", "<|im_start|>assistant"],
#sep_style=SeparatorStyle.CHATML,
sep="<|im_end|>",
stop_str="<|im_end|>",
)
)

[timothylimyl]

@noobmaster29 what happen to your PR?

[noobmaster29]

Someone on Discord mentioned that the change did not solve the issue of double EOS tokens. I have not had time to replicate it yet but I will try to look at it tomorrow.

[LZY-the-boys]

Someone on Discord mentioned that the change did not solve the issue of double EOS tokens. I have not had time to replicate it yet but I will try to look at it tomorrow.

It may be caused by dataset cache. Though the code is changed to <|im_end|>\n, the cached dataset may remain <|im_end|>\n\n and is directly loaded for train.

[noobmaster29]

Could you try a fresh process and see if it still produces double EOS and new line characters?