-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds precision to eval #148
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please wait for @abhi-mosaic approval as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good for now, and we'll debug amp_bf16
+ FSDP so we can reduce memory+compute requirements in the future.
Adds precision to eval. Sets MPT to bf16. For some reason, BF16 + FSDP requires mixed_precision: FULL. It works fine without FSDP. FP16 also works fine and gives basically the same numbers with FSDP on any setting.