Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modularize components of megablocks layer builder #1224

Merged
merged 4 commits into from
May 22, 2024

Conversation

dakinggg
Copy link
Collaborator

@dakinggg dakinggg commented May 22, 2024

Cleans up the sections of the builder functions for megablocks to be more modular, and modularizes the device mesh creation in the megablocks args.

TODO:

  • Manual test that training still works

@dakinggg dakinggg marked this pull request as ready for review May 22, 2024 07:42
@dakinggg dakinggg requested a review from mvpatel2000 May 22, 2024 07:57
Copy link
Collaborator

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but @vchiley should look as well

@mvpatel2000 mvpatel2000 requested a review from vchiley May 22, 2024 14:06
@vchiley
Copy link
Contributor

vchiley commented May 22, 2024

Manual test that training still works

Don't we have tests for MoEs (MoE testing config here)?
(I'm not against doing more testing, just asking)

@dakinggg
Copy link
Collaborator Author

@vchiley there are some simple MoE tests, I just wanted to double check e2e still works.

@dakinggg
Copy link
Collaborator Author

manual test done, merging

@dakinggg dakinggg merged commit 001e7c3 into mosaicml:main May 22, 2024
9 checks passed
@dakinggg dakinggg deleted the dm-refactor branch June 22, 2024 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants