-
Notifications
You must be signed in to change notification settings - Fork 26.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TokenizerSlow] replace_additional_special_tokens
is not doing much
#24276
Comments
cc @ydshieh since you added the feature |
But the origin of If you look the description in #20418, your code snippet does its job (although yes confusing). The |
yeah I know, but that's what it has been for years. (and I agree that the name of this introduced argument itself might be confusing too.)
If you have good idea to address the issue #20418 while reducing the (naming) confusion added in #20424, go ahead :-) (sorry, I accidentally modified your message 😭 ) |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
Closing as this is deprecated and changing the list of additional special tokens is a lot more involved than this |
Just flagging this as the
add_special_tokens
method got pretty complicated, adding a kwargs,replace_additional_special_tokens
, that supposedly can prevent replacing theself._additional_special_tokens
attribute.For any tokenizer, this will remove it from the list, but will not update the internal
trie
and thus has no effect at all:This will be addressed in #23909
The text was updated successfully, but these errors were encountered: