Update sharegpt.py

[noobmaster29]

Modified register_conv_template to resolves the double EOS token issue at the end of prompts when using Chatml template with shareGPT.py.

This issue was further discussed in the following issue:
#922 (comment)

[mhenrichsen]

Assuming this fix is correct, what should the bugged chatml order of tokens look like? Seems like the comment references an extra newline.

The correct template looks like this:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

[noobmaster29]

The issue is that when shareGPT is runned with chatml, the tokenizer adds in the default BOS and EOS tokens. In the following example, the config is as follows:

special_tokens:
bos_token: ""
eos_token: ""
unk_token: ""
tokens:

• "<|im_start|>"
• "<|im_end|>"

Tokenization results:

If the config is setup as follows:
special_tokens:
bos_token: "<|im_start|>"
eos_token: "<|im_end|>"
unk_token: ""
tokens:

• "<|im_start|>"
• "<|im_end|>"

You end up with the following tokenization (Notice that im_start and im_end are doubled at the beginning and end:

I'm not sure if this is actually an issue though. However, if conversation is not set the chatml, the tokenization is as follows (with single BOS and EOS tokens):

[winglian]

by removing this, the Conversation defaults to a sep_style=SeparatorStyle.ADD_COLON_SINGLE which results in colons getting added between the role and text.

[winglian]

Fixed w #1054