Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an example config for finetuning a 34B model on a 24GB GPU #1000

Merged
merged 2 commits into from
Dec 25, 2023

Conversation

evangriffiths
Copy link
Contributor

As stated in the README contained in the PR:

This is an example of a Yi-34B-Chat configuration. It demonstrates that it is possible to finetune a 34B model on a GPU with 24GB of VRAM.

Tested on an RTX 4090 with `python -m axolotl.cli.train examples/mistral/qlora.yml`, a single epoch of finetuning on the alpaca dataset using qlora runs in 47 mins, using 97% of available memory.

I thought this would be a useful reference config for people interested in trying out finetuning of 34B models with limited hardware.

@winglian winglian merged commit 6ef46f8 into axolotl-ai-cloud:main Dec 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants