Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust speed monitor #1645

Merged

Conversation

mvpatel2000
Copy link
Contributor

What does this PR do?

Adjusts speed monitor to start tracking at BEFORE_DATALOADER instead of BEFORE_BATCH. This resolves issues with dataloader bottlenecks being potentially masked. Note that currently it should still work since data transfer is done async, but to be safe, we should start the clock on BEFORE_DATALOADER.

@dskhudia
Copy link
Contributor

dskhudia commented Oct 18, 2022

Did you run a benchmark with it to see if it changes the numbers?

@mvpatel2000 mvpatel2000 merged commit 8ba2484 into mosaicml:dev Oct 18, 2022
@mvpatel2000 mvpatel2000 deleted the mvpatel2000/adjust-speed-monitor branch October 18, 2022 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants