Fix(tokenizer): Set rstrip,lstrip,norm to False

[NanoCode012]

Attempt to fix the tokenizer saving new special tokens properties (rstrip,lstrip) as True.

This issue can be seen in Mistral Tokenizer saved after finetuning.

[NanoCode012]

Before:

"added_tokens_decoder": {
    "0": {
      "content": "<unk>",
      "lstrip": true,
      "normalized": false,
      "rstrip": true,
      "single_word": false,
      "special": true
    },
    "1": {
      "content": "<s>",
      "lstrip": true,
      "normalized": false,
      "rstrip": true,
      "single_word": false,
      "special": true
    },
    "2": {
      "content": "</s>",
      "lstrip": true,
      "normalized": false,
      "rstrip": true,
      "single_word": false,
      "special": true
    }
  },
  "additional_special_tokens": [
    "<unk>",
    "<s>",
    "</s>"
  ],

---

After

{
  "add_bos_token": true,
  "add_eos_token": false,
  "added_tokens_decoder": {
    "0": {
      "content": "<unk>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "1": {
      "content": "<s>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "2": {
      "content": "</s>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    }
  },
  "additional_special_tokens": [
    "<unk>",
    "<s>",
    "</s>"
  ],
  "bos_token": "<s>",
  "clean_up_tokenization_spaces": false,
  "eos_token": "</s>",
  "legacy": true,
  "model_max_length": 1000000000000000019884624838656,
  "pad_token": "</s>",
  "sp_model_kwargs": {},
  "spaces_between_special_tokens": false,
  "tokenizer_class": "LlamaTokenizer",
  "tokenizer_file": "/root/.cache/huggingface/hub/models--mistralai--Mistral-7B-v0.1/snapshots/ae9d75c6b4eb39515def78c685fb4d71d49fc2cf/tokenizer.json",
  "trust_remote_code": false,
  "unk_token": "<unk>",
  "use_default_system_prompt": true,
  "use_fast": true
}

Confirmed that it's false now.

[NanoCode012]

One extra config seem to be use_fast=True. This is the default, and although Mistral's tokenizer config does not have it, Mistral's tokenizer has fast tokenizer support.

https://huggingface.co/docs/transformers/v4.34.0/en/index#supported-frameworks

[winglian]

I wonder if maybe we should only specify actual new tokens in the YML file. so remove the default tokens the tokenizer model already includes. that's probably the root of the issue.

[NanoCode012]

I wonder if maybe we should only specify actual new tokens in the YML file. so remove the default tokens the tokenizer model already includes. that's probably the root of the issue.

Yes, makes sense. We can address that in a followup PR.