Fractional epoch

[moi90]

🐛 Bug

Since upgrading to version 0.7.6, I have the following problem:

In my log, I get fractional epochs (like 0.5, 1.5, 2.5, ...) in addition to the expected integer epochs. Is this intentional? Does this mean that on_epoch_end is called twice per epoch?

In 0.7.2.dev0, this was not the case.

Expected behavior

Previously (in 0.7.2.dev0), epoch numbers in the log were integers and this is what I expect.

Environment

* CUDA:
        - GPU:
                - GeForce GTX TITAN X
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.18.5
        - pyTorch_debug:     False
        - pyTorch_version:   1.5.0
        - pytorch-lightning: 0.7.6
        - tensorboard:       2.1.1
        - tqdm:              4.46.1
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - 
        - processor:         x86_64
        - python:            3.7.6
        - version:           #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018

Additional context

I also upgraded Pytorch from 1.1.0 to 1.5.0, but this shouldn't affect the epoch counting in Lightning, right?

[williamFalcon]

Try 0.8.0rc1?

[moi90]

It happens with 0.8.1rc3 as well. After the validation pass, a half epoch is logged:

train_loss,epoch,step,datetime,val_loss,val_acc,val_f1
0.0012006619945168495,1.0,0,2020-06-20 06:18:46.498589,,,
0.0006965053617022932,1.0,50,2020-06-20 06:19:12.942534,,,
0.00047231907956302166,1.0,100,2020-06-20 06:19:40.548979,,,
0.00031291242339648306,1.0,150,2020-06-20 06:20:07.579803,,,
0.0010144615080207586,1.0,200,2020-06-20 06:20:16.187760,,,
,1.0,205,2020-06-20 06:20:16.349497,8.247037887573242,0.0,0.0
0.000277636427199468,1.5,206,2020-06-20 06:20:49.028577,,,
0.0005236843135207891,2.0,256,2020-06-20 06:21:15.270541,,,
0.0007782027823850513,2.0,306,2020-06-20 06:21:41.330511,,,
0.0003978404856752604,2.0,356,2020-06-20 06:22:07.464661,,,
0.0005978736444376409,2.0,406,2020-06-20 06:22:16.083318,,,
,2.0,411,2020-06-20 06:22:16.186468,8.181181907653809,0.0,0.0
0.0005276629235595465,2.5,412,2020-06-20 06:22:48.220048,,,
...

[moi90]

Any idea where this comes from?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.