Fix Deepspeed Zero3 Config #791
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Take away CPU Offload by default (Slows things down horribly, better off reducing batchsize), and changes LR Scheduler to a properly decaying one
fix something
|
Probably a good idea to remove CPU offload in zero2 as well. In my tests, the offloading is very minimal and if you turn it off, you get 2x training speed. |
winglian
approved these changes
Oct 27, 2023
Yea I think it is only a good idea if and when you cant fit even bs 1 on vram |
Yes, then it makes sense. Perhaps an idea is to remove it from zero2 and add a disclaimer about it in the REAMDE? |
mkeoliya
pushed a commit
to mkeoliya/axolotl
that referenced
this pull request
Dec 15, 2023
* Update zero3.json Take away CPU Offload by default (Slows things down horribly, better off reducing batchsize), and changes LR Scheduler to a properly decaying one * Update zero3.json fix something
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update DS Zero 3 conf to not use CPU Offload (because it dramatically slows things down, just reduce batchsize unless you cant fit bs 1) and replace LR Scheduler with a properly decaying one - with constant LR scheduler, it will cause loss to increase after each epoch
Compare dark pink (decaying LR Schedule) with all others, including blue, which used constant LR Schedule (the default in this conf)
