Fix amp tests #661
Merged
Fix amp tests #661
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With opt_level="O1" (the default), AMP patches many torch functions, which breaks any tests that run afterwards. This patch introduces a pytest extension that lets tests be marked with @pytest.mark.spawn so that they are run in their own process using torch.multiprocessing.spawn so that the main python interpreter stays un-patched. Note that tests using DDP already run AMP in its own process, so they don't need this annotation.
Since AMP defaults to O1 now, DP tests no longer throw exceptions. Since AMP patches torch functions, CPU inference no longer works. Skip prediction step for AMP tests.
|
Travis failure appears to be due to this unrelated issue: pytorch/vision#1718 |
Borda
approved these changes
Jan 4, 2020
| @@ -124,26 +128,6 @@ def test_amp_gpu_ddp_slurm_managed(tmpdir): | |||
| assert trainer.resolve_root_node_address('abc[23-24]') == 'abc23' | |||
| assert trainer.resolve_root_node_address('abc[23-24, 45-40, 40]') == 'abc23' | |||
|
|
|||
| # test model loading with a map_location | |||
|
tests are failing on PIL, I found somewhere that adding |
|
Supposedly torchvision will do a bug fix release early next week (0.4.2?) that will resolve the Pillow issue. The fix is already in torchvision master. I’m inclined to just wait for that and update our minimum torchvision version.
On Jan 4, 2020, at 9:40 AM, Jirka Borovec <notifications@github.com> wrote:
tests are failing on PIL, I found somewhere that adding Pollow>=4.2 solves the issue
* python-pillow/Pillow#4130<python-pillow/Pillow#4130>
* https://pillow.readthedocs.io/en/stable/releasenotes/7.0.0.html#pillow-version-constant
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#661>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAJWJZW4YFKSU5U5B25OEQTQ4CUW7ANCNFSM4KCP5IUA>.
|
|
torchvision v0.5.0 has been released with the fix:
|
|
but |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A few changes to fix AMP-related problems in our tests:
With
opt_level="O1"(the default), AMP patches many torch functions, which breaks any tests that run afterwards. This patch introduces a pytest extension that lets tests be marked with@pytest.mark.spawnso that they are run in their own process usingtorch.multiprocessing.spawnso that the main python interpreter stays un-patched.Since AMP defaults to O1 now, DP tests no longer throw exceptions. Remove the
pytest.raisesthat expects the exception.Since AMP patches torch functions, CPU inference no longer works.
Skip prediction step for AMP tests.
Note that there are still a couple of unrelated test failures that will be addressed in another PR.