Accurate Time Remaining for Training

[pertschuk]

🚀 Feature

Currently tqdm time remaining in an epoch doesn't account for the fact that validation steps are faster and so is often wildly inaccurate, fluctuating between the time remaining if all steps were validation and if all steps were training.

Motivation

Would make it much easier to see how long an epoch is gonna take without having to do the math manually.

Pitch

Could just not include validation steps in the the overall epoch tqdm iterator but that would still not be accurate, could break the epoch down into the validation chunks and show a training bar for each and then saw (1/4) if validation steps are 0.25.

Could somehow calculate the speed of validation step vs training and produce accurate times for both (not sure how difficult)...

Alternatives

Remove the time remaining from tqdm all together? If it's not accurate.

Additional context

Epoch 1:  30%|██████████████████▌                                          | 3300/10868 [10:30<24:27,  5.16it/s, loss=0.473, train_loss=0.548]

[github-actions]

Hi! thanks for your contribution!, great first issue!

[Borda]

Hey, good point, if you have some elegant solution in mind, could you send a PR?

[pertschuk]

@Borda Sure, I will look into this next week when I (hopefully) have more time

[pertschuk]

I couldn't find a good way to do with with tqdm, best temporary fix I found is to set smoothing=0 so it just takes an average of all steps rather than trying to exponentially weight most recent steps.