From 8ec9750af8297edfb74ecbe50cdb0755eaf82917 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20Robert?= Date: Thu, 29 Apr 2021 21:04:08 +0200 Subject: [PATCH 1/3] maint: cleanup test requirement files. Migrate most of their content into setup.cfg --- doc/source/developing/testing.rst | 43 ++++++++++++++---------- setup.cfg | 42 ++++++++++++++++++++++-- tests/ci_install.sh | 49 ++++++++++++---------------- tests/test_minimal_requirements.txt | 12 ------- tests/test_prerequirements.txt | 7 ---- tests/test_requirements.txt | 36 -------------------- tests/windows_conda_requirements.txt | 6 ++++ 7 files changed, 90 insertions(+), 105 deletions(-) delete mode 100644 tests/test_minimal_requirements.txt delete mode 100644 tests/test_prerequirements.txt delete mode 100644 tests/test_requirements.txt create mode 100644 tests/windows_conda_requirements.txt diff --git a/doc/source/developing/testing.rst b/doc/source/developing/testing.rst index 24145b36f10..2ff650b6971 100644 --- a/doc/source/developing/testing.rst +++ b/doc/source/developing/testing.rst @@ -74,28 +74,35 @@ run: Handling yt dependencies ^^^^^^^^^^^^^^^^^^^^^^^^ -We attempt to make yt compatible with a wide variety of upstream software -versions. However, sometimes a specific version of a project that yt depends on +Our dependencies are specified in ``setup.cfg``. Hard dependencies are found in +``options.install_requires``, while optional dependencies are specified in +``options.extras_require``. The ``full`` target contains the specs to run our +test suite, which are intended to be as modern as possible (we don't set upper +limits to versions unless we need to). The ``minimal`` target is used to check +that we don't break backward compatibility with old versions of upstream +projects by accident. It is intended to pin stricly our minimal supported +versions. The ``test`` target specifies the tools neeed to run the tests, but +not needed by yt itself. + +**Python version support.** +When a new Python version is released, it takes about +a month or two for yt to support it, since we're dependent on bigger projects +like numpy and matplotlib. We vow to follow numpy's deprecation plan regarding +our supported versions for Python and numpy, defined formally in `NEP 29 +`_. However, we try to +avoid bumping our minimal requirements shortly before a yt release. + +**Third party dependencies.** +However, sometimes a specific version of a project that yt depends on causes some breakage and must be blacklisted in the tests or a more experimental project that yt depends on optionally might change sufficiently that the yt community decides not to support an old version of that project. -To handle cases like this, the versions of upstream software projects installed -on the machines running the yt test suite are pinned to specific version -numbers that must be updated manually. This prevents breaking the yt tests when -a new version of an upstream dependency is released and allows us to manage -updates in upstream projects at our pace. - -If you would like to add a new dependency for yt (even an optional dependency) -or would like to update a version of a yt dependency, you must edit the -``tests/test_requirements.txt`` file, this path is relative to the root of the -repository. This file contains an enumerated list of direct dependencies and -pinned version numbers. For new dependencies, simply append the name of the new -dependency to the end of the file, along with a pin to the latest version -number of the package. To update a package's version, simply update the version -number in the entry for that package. - -Finally, we also run a set of tests with "minimal" dependencies installed. Please make sure any new tests you add that depend on an optional dependency are properly set up so that the test is not run if the dependency is not installed. If for some reason you need to update the listing of packages that are installed for the "minimal" dependency tests, you will need to edit ``tests/test_minimal_requirements.txt``. +**Note.** +Some of our optional dependencies are not trivial to install and their support +may vary across platforms. To manage such issue, we currently use requirement +files in additions to ``setup.cfg``. They are found in +``tests/*requirements.txt`` and used in ``tests/ci_install.sh``. How to Write Unit Tests ^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/setup.cfg b/setup.cfg index 3aef97a08f4..dd8241e8f69 100644 --- a/setup.cfg +++ b/setup.cfg @@ -37,15 +37,16 @@ project_urls = [options] packages = find: install_requires = - IPython>=1.0 + iPython>=1.0 matplotlib!=3.4.2,>=2.0.2,<3.5 more-itertools>=8.4 - numpy>=1.10.4 + numpy>=1.13.3 + pyyaml>=4.2b1 setuptools>=19.6 sympy>=1.2 toml>=0.10.2 tqdm>=3.4.0 - unyt>=2.7.2 + unyt>=2.8.0 python_requires = >=3.6 include_package_data = True scripts = scripts/iyt @@ -58,10 +59,45 @@ nose.plugins.0.10 = answer-testing = yt.utilities.answer_testing.framework:AnswerTesting [options.extras_require] +full = + astropy~=4.0.1 + cartopy~=0.18.0 + f90nml>=1.1.2 + fastcache~=1.0.2 + firefly_api>=0.0.2 + glueviz~=0.13.3 + h5py~=3.1.0 + libconf~=1.0.1 + miniballcpp>=0.2.1 + mpi4py~=3.0.3 + netCDF4~=1.5.3 + pandas~=1.1.2 + pooch>=0.7.0 + pyaml~=17.10.0 + pykdtree~=1.3.1 + pyqt5~=5.15.2 + requests~=2.20.0 + scipy~=1.5.0 + xarray~=0.16.1 hub = girder_client mapserver = bottle +minimal = + ipython==1.0.0 + matplotlib==2.0.2 + more-itertools==8.4 + numpy==1.13.3 + sympy==1.2 + unyt==2.8.0 +test = + codecov~=2.0.15 + coverage~=4.5.1 + nose~=1.3.7 + nose-exclude + nose-timer~=1.0.0 + pytest>=6.1 + pytest-xdist~=2.1.0 [flake8] max-line-length = 88 diff --git a/tests/ci_install.sh b/tests/ci_install.sh index 3c9b51d12e4..403f1935162 100644 --- a/tests/ci_install.sh +++ b/tests/ci_install.sh @@ -36,43 +36,34 @@ cp tests/matplotlibrc . # Step 1: pre-install required packages if [[ "${RUNNER_OS}" == "Windows" ]] && [[ ${dependencies} != "minimal" ]]; then - # Install some dependencies using conda (if not doing a minimal run) - CYTHON=$(grep cython tests/test_prerequirements.txt) - NUMPY=$(grep numpy tests/test_prerequirements.txt) - - CARTOPY=$(grep cartopy tests/test_requirements.txt) - H5PY=$(grep h5py tests/test_requirements.txt) - MATPLOTLIB=$(grep matplotlib tests/test_requirements.txt) - SCIPY=$(grep scipy tests/test_requirements.txt) - conda config --set always_yes yes - conda info -a - conda install --quiet --yes -c conda-forge \ - $CYTHON $NUMPY $CARTOPY $H5PY $MATPLOTLIB $SCIPY + # windows_conda_requirements.txt is a survivance of test_requirements.txt + # keep in sync: setup.cfg + while read requirement; do conda install --yes $requirement; done < tests/windows_conda_requirements.txt else python -m pip install --upgrade pip python -m pip install --upgrade wheel python -m pip install --upgrade setuptools fi -# Step 2: install required packages (depending on whether the build is minimal) +# Step 2: install deps and yt if [[ ${dependencies} == "minimal" ]]; then - # Ensure numpy and cython are installed so dependencies that need to be built - # don't error out - # The first numpy to support py3.6 is 1.12, but numpy 1.13 matches - # unyt so we'll match it here. - python -m pip install numpy==1.13.3 cython==0.26.1 - python -m pip install -r tests/test_minimal_requirements.txt + python -m pip install -e .[test,minimal] else - # Getting cartopy installed requires getting cython and numpy installed - # first; this is potentially going to be fixed with the inclusion of - # pyproject.toml in cartopy. - # These versions are pinned, so we will need to update/remove them when - # the hack is no longer necessary. - python -m pip install -r tests/test_prerequirements.txt - CFLAGS="$CFLAGS -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H" python -m pip install -r tests/test_requirements.txt -fi + # Cython and numpy are build-time requirements to the following optional deps in yt + # - cartopy + # - netcdf4 + # - pyqt5 + # The build system is however not specified properly in these projects at the moment + # which means we have to install the build-time requirements first. + # It is possible that these problems will be fixed in the future if upstream projects + # include a pyproject.toml file or use any pip-comptatible solution to remedy this. + python -m pip install numpy>=1.19.4 cython>=0.29.21 -# Step 3: install yt -python -m pip install -e . + # this is required for cartopy. It should normally be specified in our setup.cfg as + # cartopy[plotting] + # However it doesn't work on Ubuntu 18.04 (used in CI at the time of writing) + pip install shapely --no-binary=shapely + CFLAGS="$CFLAGS -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H" python -m pip install -e .[test,full] +fi set +x diff --git a/tests/test_minimal_requirements.txt b/tests/test_minimal_requirements.txt deleted file mode 100644 index af4e85d5f42..00000000000 --- a/tests/test_minimal_requirements.txt +++ /dev/null @@ -1,12 +0,0 @@ -ipython~=1.0.0 -matplotlib==2.0.2 # 2.0.0 is the first version that came out after Python 3.6, using the most recent patch in this branch as minimal -sympy~=1.2 -nose~=1.3.7 -nose-timer~=0.7.3 -pyyaml>=4.2b1 -coverage~=4.5.1 -codecov~=2.0.15 -unyt~=2.8.0 -more-itertools==8.4 -pytest~=6.1 -nose-exclude diff --git a/tests/test_prerequirements.txt b/tests/test_prerequirements.txt deleted file mode 100644 index cf5bb8782e5..00000000000 --- a/tests/test_prerequirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -# We need this file mostly because of Cartopy.. -numpy>=1.19.4 -cython>=0.29.21 - -# this is sometimes useful to avoid CI breakage when a dependency release comes out -# but some wheels (typically windows) are missing for a few hours/days -wheel diff --git a/tests/test_requirements.txt b/tests/test_requirements.txt deleted file mode 100644 index 98b64655777..00000000000 --- a/tests/test_requirements.txt +++ /dev/null @@ -1,36 +0,0 @@ -astropy~=4.0.1 -codecov~=2.0.15 -coverage~=4.5.4 -fastcache~=1.0.2 -glueviz~=0.13.3 -h5py~=3.1.0 -ipython~=7.6.1 -matplotlib<3.5,!=3.4.2 -nose-timer~=1.0.0 -nose~=1.3.7 -pandas~=1.1.2 -requests~=2.20.0 -scipy~=1.5.0 -sympy~=1.5 -pyqt5~=5.15.2 -netCDF4~=1.5.3 -libconf~=1.0.1 -shapely ---no-binary=shapely -cartopy~=0.18.0 -pyaml~=17.10.0 -mpi4py~=3.0.3 -unyt~=2.8.0 -pyyaml>=4.2b1 -xarray~=0.16.1 -firefly_api>=0.0.2 -f90nml>=1.1.2 -MiniballCpp>=0.2.1 -pooch>=0.7.0 -pykdtree~=1.3.1 -nose-exclude -more-itertools>=8.4 -tqdm>=3.4.0 -toml>=0.10.2 -pytest-xdist~=2.1.0 -pytest~=6.1 diff --git a/tests/windows_conda_requirements.txt b/tests/windows_conda_requirements.txt new file mode 100644 index 00000000000..a35a379c971 --- /dev/null +++ b/tests/windows_conda_requirements.txt @@ -0,0 +1,6 @@ +numpy>=1.19.4 +cython>=0.29.21 +cartopy~=0.18.0 +h5py~=3.1.0 +matplotlib<3.5 +scipy~=1.5.0 From c30980ff994f5d3054ed699f58848868112005b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20Robert?= Date: Fri, 7 May 2021 17:21:24 +0200 Subject: [PATCH 2/3] include doc env in setup.cfg --- setup.cfg | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/setup.cfg b/setup.cfg index dd8241e8f69..823bad3d5f3 100644 --- a/setup.cfg +++ b/setup.cfg @@ -59,6 +59,14 @@ nose.plugins.0.10 = answer-testing = yt.utilities.answer_testing.framework:AnswerTesting [options.extras_require] +doc = + alabaster + nbconvert==5.6.1 + pyregion + runnotebook + sphinx==3.1.2 + sphinx-bootstrap-theme + sphinx-rtd-theme full = astropy~=4.0.1 cartopy~=0.18.0 From 7613a037733adfd88a806db6478cd08ae637aa79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20Robert?= Date: Fri, 11 Jun 2021 19:31:51 +0200 Subject: [PATCH 3/3] maint: put cartopy specs back into a requirement file --- setup.cfg | 1 - tests/cartopy_requirements.txt | 5 +++++ 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 tests/cartopy_requirements.txt diff --git a/setup.cfg b/setup.cfg index 823bad3d5f3..a4a6e2d299b 100644 --- a/setup.cfg +++ b/setup.cfg @@ -69,7 +69,6 @@ doc = sphinx-rtd-theme full = astropy~=4.0.1 - cartopy~=0.18.0 f90nml>=1.1.2 fastcache~=1.0.2 firefly_api>=0.0.2 diff --git a/tests/cartopy_requirements.txt b/tests/cartopy_requirements.txt new file mode 100644 index 00000000000..bddb74fdffe --- /dev/null +++ b/tests/cartopy_requirements.txt @@ -0,0 +1,5 @@ +# cartopy and its dependencies are non trivial to install +# so we can't easily put these specs into setup.cfg +shapely +--no-binary=shapely +cartopy~=0.18.0