Tokenizer pad token not saved with save_pretrained

[jonathanasdf]

System Info

works on 4.33.3 (with tokenizers==0.13.3), fails on main (with tokenizers==0.14.0)

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained('stabilityai/FreeWilly2')
tokenizer.pad_token_id = 0
print(tokenizer.pad_token)
tokenizer.save_pretrained('/tmp/tok_test')
tok = transformers.AutoTokenizer.from_pretrained('/tmp/tok_test')
print(tok.pad_token)

Expected behavior

expected:

>>> print(tok.pad_token)
<unk>

actual:

>>> print(tok.pad_token)
Using pad_token, but it is not set yet.
None

[ArthurZucker]

Having a look now! Thanks

[ArthurZucker]

Okay, quite suprisingly, the previous behaviour only saved the pad token to special_tokens_map.json and not in the tokenizer_config.json. Thus the padding was set to null in both the tokenizer_config.json and the tokenizer.json. Which IMO is not good 😄

[jonathanasdf]

This particular example was fixed, thanks.

However a similar example now fails

import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained('tiiuae/falcon-40b-instruct')
tokenizer.pad_token = tokenizer.eos_token
print(tokenizer.pad_token)  # '<|endoftext|>'
tokenizer.save_pretrained('/tmp/tok_test')
tok = transformers.AutoTokenizer.from_pretrained('/tmp/tok_test')
print(tok.pad_token)  # None!

[ArthurZucker]

Thanks I'll fix this in #26570 🤗