Test user workflow in dashboard

[stefsmeets]

This PR adds a test module that tests the user workflow in the dashboard using playwright.

These tests make sure that the dashboard runs given the example data. It makes sure that the workflow completes (i.e. the RISE method produces an explanation) by checking whether the image or text is there.

These tests are somewhat time consuming, so I run them in a separate github action on python3.10/ubuntu.

Closes #567

Todo

• Add playwright to dependencies
• Add pytest marker for dashboard
• Add github action workflow (dashboard marker only)
• Skip dashboard by default: https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option

[stefsmeets]

Hi @cpranav93 @laurasootes Would either of you be willing to review this PR?

Don't mind the sonarcloud test failing, I changed 3 lines in cli.py and it complains that the test coverage on these is 0% 😬.

[laurasootes]

Hmm if I try to run pytest -v -m dashboard, the tests fail because of a connection refused error (both for LOCAL True/False). I am not sure why

I have 2 questions:

• why do you only test on ubuntu?
• where are the browsers defined that are used for the tests?

[stefsmeets]

Which OS are you running this on? I only tested it under linux. I can have a look whether it works on Windows.

My idea was that you can set LOCAL=True if you start the dashboard yourself in a separate tab. I found this convenient to be able to iterate a bit faster while developing this. Otherwise, the tests will start the dashboard in a subprocess, which is a bit slower.

1. I set up the workflow in Ubuntu because it's the most efficient runner on Github. The goal of the PR is to test the user workflow in the browser. I don't think we gain anything by testing in another OS. The functionality/correctness is verified via the other tests.
2. The browser can be defined via the --browser switch in the command line. The default is chromium, so it picks that.

[stefsmeets]

I guess these tests do not necessarily work 'out-of-the-box'. They need some fairly substantial setup, before you can use them (i.e. install the browser).

Do you think we should disable them by default, and just let github actions run them?

[laurasootes]

I am using Mac M1, and was not able to install chromium within a few minutes 😒. I think it would indeed be better to only include the test in the GitHub actions.

I agree with your reasoning on only testing on ubuntu, but would suggest to also include webkit and firefox for the browsers, just to be sure.

[stefsmeets]

Can you try again after installing chromium in playwright?

python -m playwright install chromium

Tests should now be disabled by default, so they only run in the CI or when you use:

pytest -m dashboard

Tbh, I don't think testing in firefox/webkit adds anything. The goal of this test is to test the user workflow by checking whether any explanations appear at all to avoid changes in dianna breaking the dashboard. This won't make a difference between firefox, safari or chrome in my view.

[laurasootes]

It appears that the dashboard tests are still skipped?

[stefsmeets]

It appears that the dashboard tests are still skipped?

Eep, my bad.

[stefsmeets]

Thanks for pointing that out, it's surprisingly tricky to set this up. I added a flag to tell pytest to also run the dashboard tests:

pytest --dashboard

If this flag is not specified, it will mark tests to skip if they have the dashboard mark. I hope this works now.

EDIT: still broken

[laurasootes]

Yes it works now 🥳
It runs all with the --dashboard flag, and skips the dashboard tests without it

Sounds good to only do chromium btw

Should this flag option maybe be added to the dev documentation?

[stefsmeets]

Cheers, I will add a small section to the documentation.