-
Notifications
You must be signed in to change notification settings - Fork 417
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wct save interval #3264
Wct save interval #3264
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please provide a PR description!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we were going to use train timestamp?
Working: wct-test-2m-UAMkGI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few nits
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly LGTM! Final nit :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
What does this PR do?
This PR attempts to support wall clock time as a interval metric. Users will be able to set wall clock time in training yamls to define their save interval. (https://docs.google.com/document/d/1zEs6YKN5YpnwQD-CXjyHlW7R98B3P8ORjEIqA5D8iwI/edit). Specifically, it introduces TimeUnit.SECOND, TimeUnit.MINUTE, TimeUnit.HOUR while keeping all time arithmetic in seconds (minutes and hours are converted to seconds).
manual test: wct-test-sec-v8msnl, wct-test-min-oUF2KU, wct-test-hr-A5zIE0
Checkpoints:
https://cloud.oracle.com/object-storage/buckets/axhe5a72vzpp/vincent-test/objects?region=us-phoenix-1
-->
What issue(s) does this change relate to?
N/A
Before submitting
pre-commit
on your change? (see thepre-commit
section of prerequisites)