py_env_create fails on Derecho

[samsrabin]

Brief summary of bug

As first noted by @olyson, calling ./py_env_create on Derecho gives an error related to incompatible packages. This happens even if the conda environment name is changed from ctsm_pylib to ctsm_pylib2 (or something else not matching an existing conda environment).

This can be resolved by removing version number specifiers from numpy and numba, but that may not be how we want to handle it.

General bug information

CTSM version you are using: ctsm5.1.dev163

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: n/a

Important output or errors that show the problem

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package numba-0.55.2-py37h43839f2_0 requires numpy >=1.19.5,<2.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ numba 0.55.2  is installable with the potential options
│  ├─ numba 0.55.2 would require
│  │  └─ numpy >=1.21.6,<2.0a0 , which can be installed;
│  └─ numba 0.55.2 would require
│     └─ numpy >=1.19.5,<2.0a0 , which can be installed;
└─ numpy 1.18.5  is not installable because it conflicts with any installable versions previously reported.

Trouble installing the ctsm_pylib python environment
There must be a problem in the ./python/conda_env_ctsm_py.txt conda specification environment file
Change the file and try again

[ekluzek]

This is interesting, because it shows that our long standing python environment is now out of date and non-functional. I assume this must be something that happened in the conda world, since we haven't changed our environment. I'm guessing that maybe there was a version that was removed or a dependency in an older version that was updated. But, it also shows the problems with leaving the environment unchanged for too long. I knew we would need to update the environment eventually, so I'm not completely surprised, but I also didn't know what it was going to look like for our environment aging out. Now we know this is one of the ways it looks like for this.

[ekluzek]

Oh, this is also one of the consequences of having a minimal conda environment that only gives a few requirements rather than one that is explicit and gives the exact versions for everything. The explicit version has problems as well though especially if you want to port to a different machine. I had looked into using conda-lock which had a way to work in both those ways, but I had problems with it -- but perhaps that has matured since then.

[ekluzek]

Ahh, it looks like this is due to 2dbf4ee. Dropping that it will work. So we might just need to fiddle with xesmf and numba and numpy to get this to work again. This must have worked before. So if we can figure out anything about when it worked that would be helpful..

[samsrabin]

@ekluzek It can also be resolved by removing removing version number specifiers from numpy and numba. Here are the versions of everything that results from the resulting environment solution:

pandas=1.3.5
tqdm=4.66.1
scipy=1.7.3
netcdf4=1.6.1
requests=2.31.0
packaging=23.2
numpy=1.21.6
xesmf=0.7.0
numba=0.56.3

(Plus the versions of python, xarray, pylint, and black that we specify.)

[samsrabin]

@glemieux will test the above setup with the FATES tools.

[glemieux]

For reference, looks like I recorded my notes about numpy/numba compatibility in glemieux/fates#25 (comment). TL:DR the numba=0.56.3 might be ok with the specified numpy=1.21.6. I should be able to test this out today.

[glemieux]

I was unable to create the conda environment using the above solution. I got the following errors:

Could not solve for environment specs
The following packages are incompatible
├─ hdf5 1.12.1.*  is requested and can be installed;
└─ netcdf4 1.6.1  is not installable because there are no viable options
   ├─ netcdf4 1.6.1 would require
   │  └─ hdf5 >=1.12.2,<1.12.3.0a0 mpi_mpich_*, which conflicts with any installable versions previously reported;
   ├─ netcdf4 1.6.1 would require
   │  └─ hdf5 >=1.12.2,<1.12.3.0a0 mpi_openmpi_*, which conflicts with any installable versions previously reported;
   └─ netcdf4 1.6.1 would require
      └─ hdf5 >=1.12.2,<1.12.3.0a0 , which conflicts with any installable versions previously reported.

I think this might be due to a possible typo for the netcdf4 version above. Holding only python, pylint, black, and xarray to the current tag versions results in a solution with netcdf4=1.6.0. All other versions match as noted.

From there, I was partially successful in running the FATES LUH2 datatool, although I ran into what looks like a different issue (the process just ends with Killed). I exported the environment and replicated it on my local machine and was able to successfully run the tool. As such I think this is good to go, with the one note about the netcdf version.