Bump non minimal testing infra from python 3.8 to 3.9

[neutrinoceros]

todo:

• bump requirements in order to pass build + tests stages on UNIX systems
• fix answer tests (bump them ?)
• fix windows build (currently failing because scipy 1.5.4 isn't available for win+py3.9 on conda-forge yet)
• rebase

[cphyc]

Note: according to https://github.com/numpy/numpy/releases, the first numpy version which supports Python 3.9 AND has binary wheels on all platform is numpy 1.19.3 with cython>=0.29.21. I would recommend using this combination on Python 3.9.

[neutrinoceros]

Thanks Corentin for your support. There are still a lot of failures (version conflicts) at least for the windows build. I'm going to switch it off momentarily to see wether we can at least get the builds to work on UNIX

[neutrinoceros]

Progress report:

• on Linux, the current pinned version for pyqt5 (from 2018) apparently doesn't support Python 3.9, so I needed to bump that
• as a result, some answer tests failed, so I deactivated them temporarily to see if anything else breaks
• on windows, a LOT of version conflicts appear, but I think they're really hinting at missing wheels for Python 3.9 (still too early...)
• On Mac and Linux, the build time is enormous (>13min while the minimal infra is installed in a little under 5mins). The apparent reason is that some wheels are built by pip (instead of distributed). Moreover the build fails because some wheels cannot be built at all (h5py and scipy). Maybe the pinned versions don't support python 3.9, I'll try to bump those.

Overall it seems that the environment is still not ripe for this change, but this PR is a good place to keep checking and have it ready asap.

[neutrinoceros]

note: h5py 3.0 should be compatible with Python 3.6, but it's unclear wether 3.9 is supported atm.
https://docs.h5py.org/en/latest/whatsnew/3.0.html

h5py now requires Python 3.6 or above; it is no longer compatible with Python 2.7.

I'll try bumping this later but for now I'm just bumping scipy to see what happens.

[neutrinoceros]

So bumping scipy and pyqt5 to their respective latest versions works, but maybe only because they come with wheels for python 3.9 already. It seems likely that earlier versions may be distributed in the future, but in the meantime, windows is the bottleneck here.
About answer tests: would it be reasonable to bump answers because of a pyqt5 update (and how do we check they are still acceptable) ? @Xarthisius

[Xarthisius]

About answer tests: would it be reasonable to bump answers because of a pyqt5 update (and how do we check they are still acceptable) ? @Xarthisius

Why would pyqt5 affect answers? We're using Agg backend after all.

[Xarthisius]

Whenever you need newer version of a package for py3.9 you could add the following to requirements.txt:

SomePackage~=5.1
SomePackage~=5.4; python_version > '3.8'

[neutrinoceros]

Why would pyqt5 affect answers? We're using Agg backend after all.

I don't know, but that's actually what happened on the commit I did nothing else than bumping PyQt5 !
see here
https://github.com/yt-project/yt/runs/1450535573

[neutrinoceros]

Whenever you need newer version of a package for py3.9 you could add the following to requirements.txt:

Good to know ! I'll try to avoid this as much as possible to minimise the complexity of our requirements though, but it can indeed come in handy as a last resort.

[Xarthisius]

Why would pyqt5 affect answers? We're using Agg backend after all.

I don't know, but that's actually what happened on the commit I did nothing else than bumping PyQt5 !
see here
https://github.com/yt-project/yt/runs/1450535573

Can you re-enable them? They didn't fail on jenkins with PyQt5-5.15.2. It might have been a fluke.

[neutrinoceros]

sure thing !

[neutrinoceros]

Still failing (https://github.com/yt-project/yt/pull/2978/checks?check_run_id=1453767294)

[neutrinoceros]

Actually it's just a single failure, I was expecting a lot more !

[neutrinoceros]

I opened a related issue for scipy scipy/scipy#13145

edit: now awaiting this issue conda-forge/scipy-feedstock#149

[cphyc]

That all looks reasonable to me, though I cannot tell whether the changes look are minimal or not.

[matthewturk]

So what do we think, is this controversial?

[neutrinoceros]

I cannot tell whether the changes look are minimal or not.

They probably aren't, to be honest. I find that searching manually for minimal compatible versions is terribly inefficient, but as I'm not bumping test_minimal_requirements.txt I hope this is reasonable.
In my view, test_requirements.txt's mission is to specify a working environment, not necessarily minimal nor cutting-edge. It would of course be much preferable to have deps specified as >= wherever possible but that opens the possibility for random breakage down the road in unrelated PRs, so I figure it's not worth doing this unless we deploy a bot to check that upstream updates don't break our CI (Dependabot much ?).

edit: dowstream -> upstream

[neutrinoceros]

Interestingly, pre-commit.ci failed and forgot to append a commit... reporting this upstream !

[neutrinoceros]

@yt-fido test this please