Add testbed support for web backend

[freakboy3742]

What is the problem or limitation you are having?

The web backend is not currently tested by the testbed API. We should be performing automated GUI tests of the web backend.

Describe the solution you'd like

It should be possible to run briefcase run web --test in the testbed folder and get a test and coverage report.

This should then be integrated into CI.

Describe alternatives you've considered

There isn't really an alternative here - having a test suite is a hard requirement for the web backend to become a serious Toga option.

However, there are multiple alternatives in terms of how this could be implemented.

Unfortunately, this isn't as simple as "write a probe for the web backend", as the web platform has a technical limitation. The current testbed test starts the testbed app in test mode; this immediately starts a thread, and the pytest test suite runs in the thread. This approach is needed because the pytest.run() (the entry point for pytest) is a blocking API; but in order to run any actual GUI code, the app needs to have a running app loop. Plugins like pytest-asyncio (which the testbed already uses) allow for fixtures and tests to be defined as coroutines, but pytest-asyncio expects to be able to start the event loop. That isn't an option for Toga, as the main event loop isn't a standard Python event loop - on mobile platforms, it's not even started in Python code.

This works fine for desktop and mobile platforms; but the web platform doesn't have threads. We need an alternate approach to allow for running the test suite without the use of threads.

There are (at least) 2 possible approaches that might work.

The first approach is to rework pytest so that pytest.run() is a coroutine. Testbed would start in test mode, and add pytest.run() as a task. This would allow pytest to run in the main event loop. This would have the added benefit that all the app code would be running in the GUI event loop, removing the need for workarounds like the one introduced by #2658 to avoid crashes caused by garbage collection in a non-GUI thread. However, it will likely be a complex modification to pytest, and we'd need to track changes in pytest's API.

The second approach would be to modify how we run on-platform tests. Unit testing of web code usually relies on a "remote control" interface like playwright. This allows the test suite to run on the desktop platform; the test suite starts a browser, connects to a daemon in the browser, and communicates remote control instructions to the browser (both modifying the web context, and reading the state of the web content).

In this approach, running a Toga app in "test mode" would start the testbed app with a daemon running in the event loop; that daemon opens a connection back to the test suite, the test suite passes any API calls or probe instructions back to the daemon, and the daemon returns results back to the test suite. This would require a fairly major modification to how the probe works, and how Briefcase starts tests. However, it would mean that test suites are "standard" non-async pytest suites, and wouldn't require any of the "sentinel" monitoring that is currently used by Briefcase to discover test results.

Additional context

No response

[rmartin16]

FWIW, I posed this question of approach (1) today to pytest for thoughts/ideas.

Small erratum to help avoid any potential confusion: the relevant entry point is pytest.main() instead of pytest.run() :)

[freakboy3742]

FWIW, I posed this question of approach (1) today to pytest for thoughts/ideas.

The response on that thread looks very much like what I suspected would be the case - if (1) is possible at all, it's going to require some major invasive surgery.

Small erratum to help avoid any potential confusion: the relevant entry point is pytest.main() instead of pytest.run() :)

🤦