The total number of batches shows by the progress bar of the sanity check is wrong

[manipopopo]

🐛 Bug

The total of the sanity check progress bar is set by
https://github.com/PyTorchLightning/pytorch-lightning/blob/4d0406ec8bf1c9147b34eb607411b78a9cd28243/pytorch_lightning/callbacks/progress.py#L296

The progress bar will always show trainer.num_sanity_val_steps even if the length of the validation DataLoader is less than trainer.num_sanity_val_steps.

Maybe the total could be computed by

from pytorch_lightning.trainer import data_loading

num_full_val_dataloader_batches = [
    len(dataloader) if data_loading._has_len(dataloader) else float('inf')
    for dataloader in trainer.val_dataloaders
]
self.val_progress_bar.total = convert_inf(
    sum(min(num_batches, trainer.num_sanity_val_steps)
            for num_batches in num_full_val_dataloader_batches))

We use the private function data_loading._has_len to check if dataloader has __len__, maybe we could make data_loading._has_len public.

Or we could make num_full_val_dataloader_batches (and num_full_train_dataloader_batches) a member variable of Trainer and update the value in pytorch_lightning.trainer.data_loading.TrainerDataLoadingMixin.

To Reproduce

The progress bar of the sanity check in the following code (num_sanity_val_steps == 999 and len(val_data_loader) == 10) shows

Validation sanity check:   1%|          | 9/999 [00:09<16:31,  1.00s/it]`

Code sample

import time

import pytorch_lightning as pl
from torch.utils import data


class Dataset(data.Dataset):

  def __init__(self, length):
    self._elements = list(range(length))

  def __getitem__(self, item):
    return self._elements[item]

  def __len__(self):
    return len(self._elements)


class Model(pl.LightningModule):

  def forward(self, *args, **kwargs):
    pass

  def training_step(self, *args, **kwargs):
    pass

  def train_dataloader(self):
    pass

  def configure_optimizers(self):
    pass

  def validation_step(self, *args, **kwargs):
    time.sleep(1)
    return pl.EvalResult()


if __name__ == '__main__':
  model = Model()

  val_dataset_length = 10
  val_dataset = Dataset(val_dataset_length)
  val_data_loader = data.DataLoader(val_dataset)

  trainer = pl.Trainer(num_sanity_val_steps=999, limit_val_batches=999,
                       max_epochs=0)
  trainer.fit(model, val_dataloaders=val_data_loader)

Expected behavior

The program above should be

Validation sanity check: 100%|██████████| 10/10 [00:10<00:00,  1.00s/it]

Environment

• CUDA:
  • GPU:
  • available:
  • version:
• Packages:
  • numpy: 1.18.5
  • pyTorch_debug: False
  • pyTorch_version: 1.6.0+cpu
  • pytorch-lightning: 0.9.0rc11
  • tensorboard: 1.15.0
  • tqdm: 4.48.2
• System:
  • OS: Windows
  • architecture:
    • 64bit
    • WindowsPE
  • processor:
  • python: 3.7.3
  • version: 10.0.18362

Additional context

[ananyahjha93]

@manipopopo I guess the first solution gets things done without changing anything else. Mind submitting a PR?

[rohitgr7]

I think once #2882 gets resolved this issue will be resolved automatically.

[manipopopo]

It seems that resolve #2882 will make the sanity check of Trainer(num_sanity_val_steps=5) run exactly 5 steps (or 5 * len(self.val_dataloaders) ) if the validation DataLoader has at least 5 batches.

Resolve this issue will make the total of the progress bar for the sanity check be the size of the validation DataLoader if it is less than num_sanity_val_steps (except num_sanity_val_steps == -1).

[rohitgr7]

then I would suggest fixing #2882 first in which we can make self.num_sanity_val_steps to be a list of correct num_steps for each val_dataloader and then we can just do sum(self.num_sanity_val_steps) here to assign it to progress_bar.

[manipopopo]

It seems that we have several ways to tackle the problem.

• Make self.num_sanity_val_steps a list of correct numbers of steps as @rohitgr7 suggests. ProgressBar can update self.val_progress_bar.total using self.num_sanity_val_steps. Accessing the member num_sanity_val_steps may get values different from the one passed into Trainer.__init__.

• Add a new member to save a list of correct numbers of steps or save a list of the sizes of validation DataLoaders. ProgressBar can update self.val_progress_bar.total using the new member. Users of Trainer can still get num_sanity_val_steps passed into Trainer.__init__ by accessing the member num_sanity_val_steps. (Supposing Int num_sanity_val_steps is always replaced by float limit_val_batches #2882 is resolved and the member num_sanity_val_steps is independent of limit_val_batches )

• Compute the correct number of steps in the ProgressBar with the help of pytorch_lightning.trainer.data_loading._has_len

It seems that the values of the public member Trainer.num_sanity_val_steps in stable 0.8.5 and master are different.

• In 0.8.5, Trainer.num_sanity_val_steps is num_sanity_val_steps passed in __init__.

• https://github.com/PyTorchLightning/pytorch-lightning/blob/1e68968ed7fb9b8f73df148dd48194d469655ea3/pytorch_lightning/trainer/trainer.py#L491
  sets Trainer.num_sanity_val_steps to float('inf') when the one passed in __init__ is -1.

• https://github.com/PyTorchLightning/pytorch-lightning/blob/a59e140ee814d8818d121405582133cf6b767e1a/pytorch_lightning/trainer/trainer.py#L461
  decides Trainer.num_sanity_val_steps according to limit_val_batches and num_sanity_val_steps passed in __init__.

[manipopopo]

Fixed by #2917.

[rohitgr7]

It's broken again due to refactors, opening this so that we don't forget to fix this. If required, let's fix this after a week once refactors are done.
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/trainer.py#L1256
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/evaluation_loop.py#L246
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/evaluate_loop.py#L53-L57
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/callbacks/progress.py#L341-L344

[rohitgr7]

this issue is annoying. will fix this.