Draft: Update Tokenizer Overrides Handling in models.py

[mhenrichsen]

Example:

tokenizer_overrides:
  - 28006: <|im_start|>
  - 28007: <|im_end|>

Description:

This PR introduces an enhancement to the way we handle tokenizer overrides in our models.py file.

Previously, the code did not account for the possibility of a token override not being found in the tokenizer's special tokens. This could potentially lead to silent failures where an override is skipped without any indication to the user.

The updated code now includes a check for each key in the cfg.tokenizer_overrides dictionary within the tokenizer.all_special_tokens list. If a match is found, the corresponding token's content is updated with the override value.

This change ensures that our tokenizer correctly applies all specified overrides, improving the robustness and reliability of our tokenization process.

[winglian]

thanks!! this was on my todo list too.

[NanoCode012]

Could a unit test be added for this?

[winglian]

@mhenrichsen played around with this today. The tokenizer seems to be frozen, especially for llama 3 where this would be more useful.