Metrics refactor pt1 #1411
Merged
Metrics refactor pt1 #1411
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hanlint
approved these changes
Aug 16, 2022
bandish-shah
approved these changes
Aug 16, 2022
hanlint
reviewed
Aug 17, 2022
This was referenced Aug 17, 2022
hanlint
pushed a commit
that referenced
this pull request
Aug 26, 2022
This PR implements the second half of the metrics refactor (#1411) that updates the training loop and all the Composer models. The training loop now uses the previously added state.train_metrics and state.eval_metrics from #1411 to perform all training and evaluation on the correct set of metrics. The main changes with respect to models is that the models now have split up validate() into eval_forward() and update_metrics() methods, which run an evaluation forward pass and update the metrics with the outputs of the evaluation forward pass respectively. This is mainly to get rid of the double forward pass (#467).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements part 1 of the Metrics Refactor, which entails removing metrics from the
Evaluatorclass and storing raw deep-copied metrics in theStateclass. TheEvaluatorclass now stores evaluation metric names instead of metric instances, which are matched against the model-definedmodel.val_metricsto indicate which metrics will be computed at eval time. Additionally, instead of storing all computed metrics as part ofstate.current_metrics, both the raw non-computed training and validation metrics are now stored separately as part ofstate.train_metricsandstate.eval_metrics.All tests updated and passing, regression tests will be conducted once PR 2 is ready.
Note: PR for part 2 will also touch a lot of the same code, so expect some refactors and clean up (mostly in the
Trainerclass) after this PR is merged in.Closes CO-679