Automatically find and run special tests

[carmocca]

What does this PR do?

Tired of having an outdated special_tests.sh file :)

• benchmarks special tests are also included now
• Cleaned sharded benchmark tests
• Works even if the tests are indented.
• Removes DDPLauncher

Produces the following report:

================================================================================
Ran	tests/plugins/test_deepspeed_plugin.py:183::test_invalid_deepspeed_defaults_no_precision
Ran	tests/plugins/test_deepspeed_plugin.py:198::test_warn_deepspeed_override_backward
Ran	tests/plugins/test_deepspeed_plugin.py:219::test_deepspeed_run_configure_optimizers
Ran	tests/plugins/test_deepspeed_plugin.py:249::test_deepspeed_config
Ran	tests/plugins/test_deepspeed_plugin.py:283::test_deepspeed_custom_precision_params
Ran	tests/plugins/test_deepspeed_plugin.py:304::test_deepspeed_assert_config_zero_offload_disabled
Ran	tests/plugins/test_deepspeed_plugin.py:327::test_deepspeed_multigpu
Ran	tests/plugins/test_sharded_plugin.py:261::test_ddp_sharded_plugin_test_multigpu
Ran	tests/plugins/test_rpc_sequential_plugin.py:30::test_rpc_sequential_plugin_manual
Ran	tests/plugins/test_rpc_sequential_plugin.py:54::test_rpc_sequential_plugin_manual_amp
Ran	tests/plugins/test_rpc_sequential_plugin.py:76::test_rpc_sequential_plugin_automatic
Ran	tests/plugins/test_rpc_sequential_plugin.py:100::test_rpc_sequential_plugin_with_wrong_balance
Ran	tests/plugins/test_rpc_plugin.py:68::test_rpc_function_calls_ddp
Ran	tests/callbacks/test_stochastic_weight_avg.py:117::test_swa_callback_ddp
Ran	tests/callbacks/test_pruning.py:160::test_pruning_callback_ddp
Ran	tests/models/test_sync_batchnorm.py:73::test_sync_batchnorm_ddp
Ran	tests/models/test_hooks.py:199::test_transfer_batch_hook_ddp
Ran	tests/trainer/logging_/test_train_loop_logging_1_0.py:735::test_logging_sync_dist_true_ddp
Ran	tests/trainer/test_trainer.py:1472::test_trainer_predict_dp
Ran	tests/trainer/test_trainer.py:1478::test_trainer_predict_ddp
Ran	tests/trainer/test_trainer.py:1483::test_trainer_predict_ddp_spawn
Ran	tests/trainer/test_trainer.py:1488::test_trainer_predict_1_gpu
Ran	tests/trainer/test_trainer.py:1493::test_trainer_predict_ddp_cpu
Ran	tests/trainer/optimization/test_manual_optimization.py:1073::test_step_with_optimizer_closure_with_different_frequencies_ddp
Ran	tests/trainer/optimization/test_manual_optimization.py:1148::test_step_with_optimizer_closure_with_different_frequencies_ddp_with_toggle_model
Ran	tests/trainer/test_data_loading.py:101::test_replace_distrubuted_sampler_custom_dataloader_custom_batch_sampler
Ran	tests/utilities/test_all_gather_grad.py:50::test_all_gather_collection
Ran	tests/utilities/test_all_gather_grad.py:100::test_all_gather_sync_grads
Ran	tests/test_profiler.py:300::test_pytorch_profiler_trainer_ddp
Skipped	tests/test_profiler.py:431::test_pytorch_profiler_nested_emit_nvtx
Ran	tests/checkpointing/test_checkpoint_callback_frequency.py:109::test_top_k_ddp
Ran	tests/accelerators/test_multi_nodes_gpu.py:35::test_logging_sync_dist_true_ddp
Ran	tests/accelerators/test_multi_nodes_gpu.py:75::test__validation_step__log
================================================================================

Fixes https://github.com/PyTorchLightning/internal-dev/issues/125

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• [n/a] Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• [n/a] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

[codecov]

Codecov Report

Merging #6669 (c64578e) into master (9be092d) will decrease coverage by 8%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #6669     +/-   ##
========================================
- Coverage      91%     83%     -8%     
========================================
  Files         192     193      +1     
  Lines       12248   12981    +733     
========================================
- Hits        11162   10771    -391     
- Misses       1086    2210   +1124     

[carmocca]

One test seems to be failing even though it's shown as Passed:

Traceback (most recent call last):
  File "/__w/1/s/pytorch_lightning/trainer/trainer.py", line 599, in run_train
    self.train_loop.run_training_epoch()
  File "/__w/1/s/pytorch_lightning/trainer/training_loop.py", line 547, in run_training_epoch
    epoch_output, self.checkpoint_accumulator, self.early_stopping_accumulator, self.num_optimizers
  File "/__w/1/s/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 442, in log_train_epoch_end_metrics
    self.training_epoch_end(model, epoch_output, num_optimizers)
  File "/__w/1/s/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 494, in training_epoch_end
    epoch_output = model.training_epoch_end(epoch_output)
  File "/__w/1/s/tests/utilities/test_all_gather_grad.py", line 70, in training_epoch_end
    assert gathered_loss["losses_tensor_int"][0].dtype == torch.int64
AssertionError: assert torch.int32 == torch.int64
  +torch.int32
  -torch.int64

https://dev.azure.com/PytorchLightning/pytorch-lightning/_build/results?buildId=4295&view=logs&j=3afc50db-e620-5b81-6016-870a6976ad29&t=d9f671c5-a304-5675-5394-961fd7f98b9b&l=3644

cc: @tchaton as the failing test author

edit: resolved

[pep8speaks]

Hello @carmocca! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-03-26 16:41:50 UTC

[awaelchli]

Nice to see this automated. This will be useful for new contributors who don't know how these special tests work.

[SeanNaren]

I've seen this pattern in other repos, hopefully we can get this fix into other peoples' CI!

Fantastic work @carmocca :)