[Wav2Vec2] Fix tokenizer set lang

[sanchit-gandhi]

What does this PR do?

The PR #23909 removed unique_no_split_tokens as an attribute of the Wav2Vec2 tokenizer, however it is required to set the language in the method .set_target_lang:

transformers/src/transformers/models/wav2vec2/tokenization_wav2vec2.py

Line 237 in dcbfd93

self.unique_no_split_tokens.append(token)

Thus, calling .set_target_lang currently throws an error:

from transformers import Wav2Vec2CTCTokenizer

tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("facebook/mms-1b-all")
tokenizer.set_target_lang("spa")

Output:

Traceback (most recent call last):
  File "/Users/sanchitgandhi/transformers/debug_tokenizer.py", line 4, in <module>
    tokenizer.set_target_lang("spa")
  File "/Users/sanchitgandhi/transformers/src/transformers/models/wav2vec2/tokenization_wav2vec2.py", line 237, in set_target_lang
    self.unique_no_split_tokens.append(token)
AttributeError: 'Wav2Vec2CTCTokenizer' object has no attribute 'unique_no_split_tokens'

This PR re-instates unique_no_split_tokens as an attribute of the tokenizer. Note that this should already be tested for in the following test:

transformers/tests/models/wav2vec2/test_tokenization_wav2vec2.py

Line 798 in dcbfd93

tokenizer.set_target_lang("spa")

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[andergisomon]

My space suddenly had a runtime error because of this.

[sanchit-gandhi]

Sorry to hear that @andergisomon - note that running transformers on the latest PyPi version should bypass this error. It's only on main at the moment:

pip uninstall transformers -y
pip install --upgrade transformers

[ArthurZucker]

Thanks, I'd rather remove the addition of these tokens to unique_no_split_tokens as they won't be used.( #26322 was on it's way)

The issue is that adding them to self.unique_no_split_tokens just does nothing. A fix that would keep the previous behaviour is to rather do self.add_tokens(self.encoder.keys()). It's a bit inefficient but will make sure they are not split!

[sanchit-gandhi]

Thanks for the review! Applied the suggestions from your comment - look good to you?

[ArthurZucker]

Let's simplify a bit and good to go

[ArthurZucker]

add_tokens loops over the tokens, so I'd recommend adding them using:

Suggested change

	for token in self.encoder.keys():
	if len(token) > 1:
	self.unique_no_split_tokens.append(token)
	self.add_tokens(token)
	self.add_tokens(self.encoder.keys())

as an internal checks makes sure let(token)>1

[sanchit-gandhi]

But we need lstrip=rstrip=True here, so need to use AddedToken(lstrip=True, rstrip=True) as we had before. If that's ok with you, I'll merge

[ArthurZucker]

Fine with me!