Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer' #124

Open
DuFangY opened this issue Oct 7, 2023 · 7 comments
Open

Comments

@DuFangY
Copy link

DuFangY commented Oct 7, 2023

No description provided.

@JimmyJIA-02
Copy link

+1, is that solved?

@JimmyJIA-02
Copy link

alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved.

@DuFangY
Copy link
Author

DuFangY commented Oct 11, 2023

alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved.
I solved the problem by following your method, thank you!

@shituo123456
Copy link

4.27.1

thank you,solved

@dingqiangliu
Copy link

move line "self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) " before "super().init(" at "init" of class ChatGLMTokenizer can solve this issue.

@Yancy456
Copy link

check if you are using ChatGLM1 but no ChatGLM2

@ovjust
Copy link

ovjust commented Jan 24, 2024

4.27.1 not run on my computer.

Exception has occurred: RuntimeError
Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 22, in init
self.sp.Load(model_path)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 64, in init
self.text_tokenizer = TextTokenizer(vocab_file)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 221, in init
self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
File "D:\1MyFiles\code\python\visualglm-6b-main\2test.py", line 4, in
tokenizer = AutoTokenizer.from_pretrained(r"D:\1MyFiles\code\python\visualglm-6b-model-gitee", trust_remote_code=True)
RuntimeError: Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants