Correct @is_flaky test decoration

[qubvel]

What does this PR do?

Some tests are marked with @is_flaky decorator without brackets, which leads to always passing the test even if an exception is always raised in the test.

For example:

@is_flaky  # <------ should be @is_flaky()
def test_flash_attn_2_equivalence(self):
    ...

P.S. Fixing this may lead to failing flaky tests, but now these tests are just broken.

Who can review?

@ydshieh

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Thank you for identifying this!

[ydshieh]

Let's trigger a CI that runs slow tests for a few models involved.

I have put a label run_slow, could you push an empty commit with [run_slow] gemma, xxx, yyy, where xxx, yyy are some model names that you can think of relevant.

[qubvel]

@ydshieh I triggered slow tests, but it seems like flash attention is not installed in CI (according to pip freeze output) -> tests were skipped:

[ydshieh]

oh, right. Those flash attn tests will only run with .github/workflows/push-important-models.yml which is triggered when a PR being merged to main.

One last thing could try is to use

https://github.com/huggingface/transformers/actions/workflows/ssh-runner.yml

Internal notion doc: https://www.notion.so/huggingface2/SSH-into-our-runners-d9c24de77d0843b2a308f28171106001

It's more a good chance to learn how to use this. Would you like to give it a try even it's somehow manual?

For docker image, you will have to fill huggingface/transformers-all-latest-gpu

You will have to do (once inside the runner)

cd transformers

and

          apt install sudo
          sudo -H pip install --upgrade pip
          sudo -H pip uninstall -y transformers 
          sudo -H pip install -U -e ".[testing]" 
          MAX_JOBS=4 pip install flash-attn --no-build-isolation

[ydshieh]

will merge after an approval from @amyeroberts (as a manual run inside the runner is done .)

[qubvel]

Gemma FA2 tests

[amyeroberts]

Thanks for fixing and running tests!