[feat] OSS flatten state dict #65
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
cc @mannatsingh |
msbaines
reviewed
Sep 9, 2020
blefaudeux
commented
Sep 9, 2020
| @@ -119,7 +119,7 @@ def closure(): | |||
| print(f"[{dist.get_rank()}] : Mean speed: {mean:.2f} +/- {std:.2f}") | |||
|
|
|||
| if use_oss and check_regression and dist.get_rank() == 0: | |||
| assert (mean - 3.0 * std) < reference_speed, "Speed regression detected" | |||
| assert (mean + 3.0 * std) > reference_speed, "Speed regression detected" | |||
There was a problem hiding this comment.
this was wrong, fixed in another PR but might as well be fixed here (I was bumping into this locally)
There was a problem hiding this comment.
(we want the speed to increase, not decrease.. the test initially made sense when comparing runtime, now that it compares frames per second higher is better)
blefaudeux
commented
Sep 9, 2020
| parser.add_argument("--check_regression", action="store", default=True, type=bool) | ||
| parser.add_argument("--reference_speed", action="store", default=39.82, type=float) | ||
| parser.add_argument("--check_regression", action="store_true", default=False) | ||
| parser.add_argument("--reference_speed", action="store", default=32.32, type=float) |
There was a problem hiding this comment.
39 was the default speed for SGD, I had changed that earlier to RMSProp when checking for the memory pressure, somehow this change was lost
msbaines
approved these changes
Sep 9, 2020
myleott
pushed a commit
that referenced
this pull request
Feb 22, 2021
* add unit test pack/unpack kwargs * added two more corner cases * more doc and more tests * more corner cases * formatting * Update fairscale/utils/containers.py Co-authored-by: Sam Shleifer <sshleifer@gmail.com> * with pytest.raises is awesome * addressed comment * add tuple to be tested Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
Changes the structure of the returned state dict with respect to the param_groups to make it closer to what a vanilla optimizer would return (un-shard them). Shard again when loading
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃