Technical debt with deps specifications

[neutrinoceros]

Bug report

Bug summary

Currently, there are several files specifying constraints for our deps. I'll try to list them all here with short descriptions

• setup.py: defines mandatory runtime dependencies (update: migrated to setup.cfg in Migrate metadata from setup.py to setup.cfg #3078)
• pyproject.toml: defines only Cython and numpy as compile-time deps
• tests/test_prerequirements.txt: again, constrains for Cython and numpy, but this time it's needed because Cartopy (an optional dep to us) doesn't define its own compile time deps correctly (?)
• tests/test_requirements.txt: contains all optional deps of yt and (mostly) up-to-date versions of mandatory deps
• tests/ci_install.sh: contains "minimal" requirements for Cython and numpy (for Cartopy again)
• tests/test_minimal_requirements.txt: defines minimal mandatory runtime deps

Evidently this is getting pretty hard to maintain, especially in PRs like #2917 (ok maybe the plural isn't appropriate)... and quite confusing for everyone.

It seems to me that, for the most part (i.e. ignoring the Cartopy Problem™️), the problem can be addressed by using _only_™️ setup.cfg (or setup.py) to specify minimal runtime deps

# this illustrates the syntax, content is irrelevant
[options]
install_requires =
    setuptools>=19.6
    matplotlib>=1.5.3; python_version < 3.0
    sympy>=1.2
    numpy>=1.10.4
    IPython>=1.0
    unyt>=2.7.2
    more_itertools>=8.4
    tqdm>=3.4.0
    toml>=0.10.2

and a "full" env could be specified as

[options.extras_require]
# again, content is irrelevant
full =
    requests>=2.20.0
    netCDF4>=1.5.3
    ...

though I admit it is not clear to me if this would still work with the most recent versions of pip.
Note that the identifier "full" is purely arbitrary.

The Cartopy Problem ™️

It seems a lot of the technical debt described above is due to the fact Cartopy isn't trivial to install for non-conda environments. However our own tests are run with conda so the setup could still be simplified (I think).

Moreover, we have this helpful comment

Getting cartopy installed requires getting cython and numpy installed
first; this is potentially going to be fixed with the inclusion of
pyproject.toml in cartopy.

and I'm glad to announce that this commit SciTools/cartopy@f6aadf8 (⚠️ not released yet) apparently solved the issue upstream, so there's room for experimentation and simplification on our side !

Issue with PEP 517 (pyproject.toml)

Our own pyproject.toml is causing some undesirable side effects lately because it specifies compile-time dependencies that are installed in isolation, so they actually ignore the runtime environment. This is problematic because compiling yt with e.g. numpy 1.20.0 while 1.19.0 is installed will create a faulty install where import yt raises an error.

I honestly don't know how to resolve this properly, the only workaround I know atm is to advise users to upgrade numpy before compiling yt (which is done here #3073).

Useful References

• setuptools: specifying optional dependencies
• Installing Cartopy
• gotchas with the newly released pip dependencies resolver
• Adressing a common misconception about setup.py VS requirements.txt

[cphyc]

Thanks so much for this! I do agree we have too many ways to define our dependencies and I cannot see one that you would have missed. I'd be happy to see everything be moved to one single file (e.g. setup.cfg or setup.py). In particular, this would allow to install the dependencies using the syntax pip install yt[full], which I find quite clean.

One thing I am wondering about is how to handle compile-time dependencies? Should this stay in pyproject.toml?

[neutrinoceros]

One thing I am wondering about is how to handle compile-time dependencies? Should this stay in pyproject.toml?

atm I don't think we can avoid this yeah, but at least the separation is quite clean and documenting this state would already reduce the amount of headache it's currently generating, I think.

[neutrinoceros]

Another note is that I've been experimenting with this pre-commit hook (https://github.com/asottile/setup-cfg-fmt) on small projects and it is a very nice helper to maintain consistency in setup.cfg (in particular, it can auto-discover and update [options] python_requires). This provides some incentive to migrate the metadata from setup.py to setup.cfg.
The hook however is extremely long to run (several seconds on very small projects), so I'll need to further experiment with it to know if scales to the size of yt.

[neutrinoceros]

and as pointed out by @cphyc , there's a tool to automate the migration from setup.py to setup.cfg https://github.com/asottile/setup-py-upgrade

edit: it seems this can't be done because some functionalities described in our setup.py can not be translated to setup.cfg. Namely distclass, ext_modules and cmdclass. Maybe there's a way to keep them in setup.py while still migrating most of the data to setup.cfg but this looks doubtful at best.

edit again: doable after all #3078

[neutrinoceros]

This is mostly solved with #3236 merged. There is still room for improvement but nothing really obvious atm.