-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how can I replace tokenizer TinyChart #94
Comments
Hi @LilDevsy0117, |
Thanks @zhangliang-04 train.sh and modified train.py config = LlavaConfig.from_pretrained(model_args.model_name_or_path) Tokenizer, init_tokenizer = TokenizerSelect('synatra')() However, I encountered the following error. You are using a model of type llava to instantiate a model of type tiny_chart_synatra. This is not supported for all configurations of models and can yield errors. WARNING: tokenization mismatch: 203 vs. 210. (ignored) |
I want to change the tokenizer so that it can be applied to Korean
I would appreciate it if you could change LLM_PATH and additionally let me know which part of the code should be modified.
The text was updated successfully, but these errors were encountered: