Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[testing] skip decorators: docs, tests, bugs #7334

Merged
merged 7 commits into from
Sep 23, 2020

Conversation

stas00
Copy link
Contributor

@stas00 stas00 commented Sep 23, 2020

This PR:

  • fixes a bug in require_torch_and_cuda
  • makes all skip decorators consistent code-wise
  • adds a test for testing combinations of skip decorators and other decorators
  • clarifies testing.rst notes

OK, so other than a small bug in require_torch_and_cuda our skip decorators can be used in any order.

The only problem I found so far is when they are used together with @parameterized, which has to come first and skip decorators last. It rewrites test names, to create a unique test name for each parameter group. and then it runs them - it has no idea it may have any skip decorators before it (The decorators get all stacked, and one below has no idea what the one above does).

If you find other unusual decorators, please let me know and I will investigate.

Partially fixes #7326

@LysandreJik, @sgugger

@stas00 stas00 mentioned this pull request Sep 23, 2020
@codecov
Copy link

codecov bot commented Sep 23, 2020

Codecov Report

Merging #7334 into master will decrease coverage by 3.75%.
The diff coverage is 43.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7334      +/-   ##
==========================================
- Coverage   80.48%   76.73%   -3.76%     
==========================================
  Files         181      181              
  Lines       34827    34827              
==========================================
- Hits        28032    26724    -1308     
- Misses       6795     8103    +1308     
Impacted Files Coverage Δ
src/transformers/testing_utils.py 67.28% <43.75%> (-1.24%) ⬇️
src/transformers/configuration_reformer.py 21.62% <0.00%> (-78.38%) ⬇️
src/transformers/modeling_reformer.py 16.87% <0.00%> (-77.64%) ⬇️
src/transformers/modeling_tf_xlm.py 19.02% <0.00%> (-74.21%) ⬇️
src/transformers/modeling_tf_lxmert.py 22.14% <0.00%> (-72.41%) ⬇️
src/transformers/modeling_marian.py 60.00% <0.00%> (-30.00%) ⬇️
src/transformers/tokenization_openai.py 71.21% <0.00%> (-12.88%) ⬇️
src/transformers/activations.py 81.81% <0.00%> (-4.55%) ⬇️
src/transformers/modeling_bart.py 93.93% <0.00%> (-0.34%) ⬇️
src/transformers/tokenization_utils_base.py 93.64% <0.00%> (-0.28%) ⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25b0463...c17c310. Read the comment docs.

@stas00 stas00 changed the title [testing] skip decorators: docs, tests, bugs [wip] [testing] skip decorators: docs, tests, bugs Sep 23, 2020
@stas00 stas00 changed the title [wip] [testing] skip decorators: docs, tests, bugs [testing] skip decorators: docs, tests, bugs Sep 23, 2020
Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks @stas00!

if torch_device != "cuda":
return unittest.skip("test requires CUDA")
return unittest.skip("test requires CUDA")(test_case)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@LysandreJik LysandreJik merged commit 28cf873 into huggingface:master Sep 23, 2020
@stas00 stas00 deleted the skip branch September 23, 2020 15:22
thevasudevgupta pushed a commit to thevasudevgupta/transformers that referenced this pull request Sep 29, 2020
* skip decorators: docs, tests, bugs

* another important note

* style

* bloody style

* add @pytest.mark.parametrize

* add note

* no idea what it wants :(
fabiocapsouza pushed a commit to fabiocapsouza/transformers that referenced this pull request Nov 15, 2020
* skip decorators: docs, tests, bugs

* another important note

* style

* bloody style

* add @pytest.mark.parametrize

* add note

* no idea what it wants :(
fabiocapsouza added a commit to fabiocapsouza/transformers that referenced this pull request Nov 15, 2020
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants