-
Notifications
You must be signed in to change notification settings - Fork 26.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GGUF: Fix llama 3 GGUF #31358
GGUF: Fix llama 3 GGUF #31358
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool! Missing some small stuff but overall LGTM
decoders.Fuse(), | ||
decoders.Replace("▁", " "), | ||
] | ||
if not self.uses_byte_level_decoding: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pre_tokenizer should alsobe ByteLevel
@@ -121,7 +121,11 @@ def __init__(self, *args, **kwargs): | |||
gguf_param = load_gguf_checkpoint(kwargs.get("vocab_file")) | |||
architecture = gguf_param["config"]["model_type"] | |||
tokenizer_dict = gguf_param["tokenizer"] | |||
fast_tokenizer = convert_gguf_tokenizer(architecture, tokenizer_dict) | |||
fast_tokenizer, additional_kwargs = convert_gguf_tokenizer(architecture, tokenizer_dict) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah sound good to me TBH
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Last few nits!
self.additional_kwargs["eos_token"] = bos_token | ||
self.additional_kwargs["bos_token"] = eos_token | ||
|
||
if self.uses_byte_level_decoding: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flag might be a bit miss-leading, since it's byte level encoding !
decoders.Replace("▁", " "), | ||
] | ||
if not self.uses_byte_level_decoding: | ||
sequence = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you can have bytefallback along with byteLevel decoding!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is misleading indeed, i changed a bit the arg name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still wrong! the sequence should always be bytefallbacl fuse and replace, but if you are a llama3, you add the bytelebel as well
|
||
# This is tricky as the additional kwargs are passed after legacy is force-set in LlamaTokenizer's | ||
# init. | ||
tokenizer.normalizer = normalizers.Sequence([]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit tricky, additional_kwargs are created in this call:
super().__init__( |
legacy=True
is passed. additional_kwargs
is taken into account after the init of the child class here: super().__init__(**kwargs) |
legacy
is silently ignored :/ I had to come up with the hack you shared offline
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM left a last nit!
decoders.Replace("▁", " "), | ||
] | ||
if not self.uses_byte_level_decoding: | ||
sequence = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still wrong! the sequence should always be bytefallbacl fuse and replace, but if you are a llama3, you add the bytelebel as well
@@ -171,6 +173,19 @@ def test_qwen2_q4_0(self): | |||
EXPECTED_TEXT = "Hello.jsoup\n\nI am a beginner" | |||
self.assertEqual(tokenizer.decode(out[0], skip_special_tokens=True), EXPECTED_TEXT) | |||
|
|||
def test_llama3_q4_0(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The string that you are testing could be improved! Specifically to take into account the special tokens and etc.
But alright otherwise!
* Create push-important-models.yml * llama3 support for GGUF * fixup * Update src/transformers/integrations/ggml.py * fix pre-tokenizer * fix * fix * fix * fix * fix * fix * address final comment * handle special tokens + add tests
What does this PR do?
Replaces: #31215
Fixes: #30391 (comment)
Currently it is not possible to load Llama-3 GGUF models due to the fact that the Llama3 tokenizer is slightly different from the previous Llama models.
A way to detect that we are having a llama-3 gguf model is to check for the attribute
tokenizer.model
(registered astokenizer_type
inproto
) of the GGUF file (see an example below taken from this checkpoint).Firstly, the GGUF file directly contains the merges attributes, in that case,
scores
is not present anymore so creating a dummy scores array is sufficient and avoids errors.Secondly, I addressed the case where
unkonwn_token_id
is registered within the proto object ( seems to be the case for some Llama2 checkpoints)In addition to that, note that llama3 uses different special tokens that are different from the default special tokens of LlamaTokenizer (e.g. it uses
<begin_of_text>
instead of<s>
). This can be fixed by passing the correct special tokens to the tokenizers init method - therefore we need a logic to pass these new args / kwargs to that method. I propose to create a new attributeadditional_kwargs
inside the tokenizer converter and pass that along to the conversion logic. Moreover, I made sure that the special tokens handling is more "universal" (in the past we were hardcoding everything)decoders.ByteLevel
to properly decode the generated tokens as spaces are encoded with the characterĠ
in the Llama3 tokenizer - this is not the case for Llama 1 & 2 tokenizers.Finally I added a new test that uses a Llama3 checkpoint and I made sure previous tests all pass