-
-
Notifications
You must be signed in to change notification settings - Fork 780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added tiny llama examples for lora and qlora #1027
Conversation
examples/tiny-llama/qlora.yml
Outdated
special_tokens: | ||
bos_token: "<s>" | ||
eos_token: "</s>" | ||
unk_token: "<unk>" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
special_tokens: | |
bos_token: "<s>" | |
eos_token: "</s>" | |
unk_token: "<unk>" | |
special_tokens: |
these are un-necessary
examples/tiny-llama/lora.yml
Outdated
special_tokens: | ||
bos_token: "<s>" | ||
eos_token: "</s>" | ||
unk_token: "<unk>" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
special_tokens: | |
bos_token: "<s>" | |
eos_token: "</s>" | |
unk_token: "<unk>" | |
special_tokens: |
can you also remove the existing tiny-llama yml here too pls? https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/examples/llama-2/tiny-llama.yml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Awesome! |
example for TinyLlama1.1B using lora and qlora. Completes in roughly 10 minutes on a 4090.