Refactor updates #2
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The _FillValue in the NLDAS params needs to be explicity changed to the globalNdv value so the masked areas can be reconstructed on the other side of an MPI broadcast
* Re-include removed HRRR precipitaion for use in Short-Range forecasts * Fix issue with MRMS missing data when used as an overlay in AnA
* change negative value replacement from 0 to config_options.globalNdv * remove log warning (since negatives are most likely missing values in radar-only files)
Removed line continuations ('\') in the "Set up ESMF and wgrib2" run step. They were contributing to the CI failures.
…ad config_data symlinks and config_data folder. Now if the folder doesn't exist it triggers the download to populate it. Added hook to allow either esmpy or ESMF python library name (name changed in ESMPy v 8.4.0)
… Updated CI test to run tests out of a docker container instead of building everything as part of the workflow (building ESMF was taking > 20 minutes). Container name is currently andygaydos/wrf_hydro_forcing - this will need to be changed to point to final location of docker image.
… on local repo clone. Added propagation of exit status to python scripts
…r produce errors. Additional YAML key checks (using try/except blocks). Updated ioMod.py to remove temporary grib2.tbl file from scratch dir when finished with it. Uncommented NETCDF variable definition, as it was causing 'not defined' errors
…rocesses. Added pytest-mpi to requirements.
….py to test various MPI n_procs (none, 1, 4, and 8) for consistency
…t, and indexing an array with a string instead of a number). Reworked test fixtures to more easily test outputs of various configurations (time of day, day of year, AnA vs forecast)
…lated to CFS-specific parameters in config.py and bias_correction.py
* Regridding subclasses with abstract class * New YAML configuration changes * Revert changes to AnA metadata handling * removed July 2022 changes to AnA metadata handling and reverted to v2.1 behavior. * removed unneeded special-case code to artificially limit Alaska AnA cycle/forecast counting * AK Short-Range NBM support (NCAR#90) * Shift NBM cycles back 3 hours for AK Short-Range to match HRRR * Fix unintentional Alaska MR NBM shift * Onboarding updates * Fix parallel bug in NBM regridding * Fix incorrect time divisor in AK ExtAnA * Add spatial metadata globals to output files (NCAR#94) * Support NBM as primary forcing input (NCAR#93) * Additional NBM support * Add support for NBM input with external terrain file * Add creep-fill for NBM * Remove precipitation from CONUS HRRR forcing (NCAR#92) * Resolved merge conflicts with the main branch * Create README.md * Change `model_total_valid_times` attrib to int32 * Change globalNdv to -999999 * Add to ignored global metadata * Don't mask NLDAS params when reading netCDF var The _FillValue in the NLDAS params needs to be explicity changed to the globalNdv value so the masked areas can be reconstructed on the other side of an MPI broadcast * Adding pytest unit tests and related changes * Adding change to python version in the workflow * Adding change to requirements file by removing ESMpy in the workflow * Adding change to requirements file by removing unnecessary packages in the workflow * Adding change to requirements file by removing unnecessary packages in the workflow * Adding change to requirements file by removing unnecessary packages in the workflow * Adding change to requirements file by removing unnecessary packages in the workflow * Adding change to python version * Adding change to linux system type * Adding change to linux system type * Adding change to linux system type * Adding change to linux system type * Adding change to linux system type * Adding change to linux system type * Removed disutils from requirements * Removed packages from requirements * Removed packages from requirements * Removed packages from requirements * Removed packages from requirements * Removed packages from requirements * Removed packages from requirements * Adding python path dependencies * Adding python path dependencies * Adding python path dependencies * Adding python path dependencies * Adding WrfHydroForcing before core in import statements * Removing WrfHydroForcing before core in import statements * change to python path * change to python path * change to setup * change to setup * change to setup * python 3.8 * python 3.8 * python 3.8 * python 3.8 * python 3.8 * Restructured the tests and added relative paths * Added conda init to github workflow * Modified conda init to github workflow * Modified conda init to github workflow * Modified conda activate to source in github workflow * Modified conda activation in a non interactive terminal in github workflow * Added Installed /opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/site-packages/wrf_hydro_mfe-1.0-py3.8.egg in github workflow * Added ESMPy install * Added ESMPy install * Added ESMPy install * Added ESMPy install * Added ESMPy install * Added ESMPy install * Delete configOptions_empty.yaml * Delete configOptions_invalid1.yaml * Delete configOptions_invalid2.yaml * Delete configOptions_missing.yaml * Delete configOptions_missingKey.yaml * Delete configOptions_valid.yaml * Delete inputForcings_empty.yaml * Delete inputForcings_invalid.yaml * Delete inputForcings_missing.yaml * Delete inputForcings_valid.yaml * Delete report.html * Fix issues with MRMS overlaying HRRR/RAP * Re-include removed HRRR precipitaion for use in Short-Range forecasts * Fix issue with MRMS missing data when used as an overlay in AnA * Set negative MRMS values to globalNdv silently * change negative value replacement from 0 to config_options.globalNdv * remove log warning (since negatives are most likely missing values in radar-only files) * Update tests.yml Removed line continuations ('\') in the "Set up ESMF and wgrib2" run step. They were contributing to the CI failures. * Updated ESMF_7_1_0r source URL (now hosted on Github) * Wrong spelling of ESMF tarball filename. * More paths fixed in tests.yml * More tests.yml fixes in conda setup * Added ubuntu install of mpich * Argh. Need sudo for installing packages on runner machine * Trying newer version of ESMF * More mods for building ESMFpy and wgrib2 * wgrib2 and ESMF working, trying to get pytest working * Added downloading of test data from google drive to tests. Removed dead config_data symlinks and config_data folder. Now if the folder doesn't exist it triggers the download to populate it. Added hook to allow either esmpy or ESMF python library name (name changed in ESMPy v 8.4.0) * Changed np.int to int * Added run_tests.py to download test data first before running pytest. Updated CI test to run tests out of a docker container instead of building everything as part of the workflow (building ESMF was taking > 20 minutes). Container name is currently andygaydos/wrf_hydro_forcing - this will need to be changed to point to final location of docker image. * Changed URL for gdrive download. Added run step to change permissions on local repo clone. Added propagation of exit status to python scripts * Bug fix * Whoops gdrive mod didn't work... * Changed docker image tag name to wrfhydro/wrf_hydro_forcing:latest * Changed np.int and np.bool to int and bool respectively, as the former produce errors. Additional YAML key checks (using try/except blocks). Updated ioMod.py to remove temporary grib2.tbl file from scratch dir when finished with it. Uncommented NETCDF variable definition, as it was causing 'not defined' errors * Added initial mpi tests. Modified run_tests.py to use mpirun with 4 processes. Added pytest-mpi to requirements. * Added tests and assertions for scattering an array using ESMF and mpi * Added geoMod tests. Added mpi scatter_array tests. Modified run_tests.py to test various MPI n_procs (none, 1, 4, and 8) for consistency * Forgot to add fixtures.py - a common set of pytest test fixtures * Added bias correction tests * Fixed bug in bias_correction.py (variable referenced before assignment, and indexing an array with a string instead of a number). Reworked test fixtures to more easily test outputs of various configurations (time of day, day of year, AnA vs forecast) * Added bias correction tests for CFS. Fixed yaml configuration bugs related to CFS-specific parameters in config.py and bias_correction.py --------- Co-authored-by: ishita9 <ishitas@ucar.edu> Co-authored-by: ishita9 <36771676+ishita9@users.noreply.github.com> Co-authored-by: ishita9 <ishita9@github.com> Co-authored-by: Ryan Cabell <rcabell@ucar.edu> Co-authored-by: Andrew Gaydos <gaydos@derecho4.hsn.de.hpc.ucar.edu>
…n tests to shared fixtures.py file
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Development of the Github Actions CI tests. I've created a pre-built docker container with the base libraries (ESMF, mpich, etc) preinstalled as they were taking a really long time to build during the CI tests.
Upgraded the library versions in the docker container:
ESMF (8.5.0), mpich (4.), python (3.11) and wgrib2 (3.1.2)Moved the config_data from this repo (was just symlinks) and instead download the tarball from google drive when running the pytests (when launched from
core/tests/run_tests.py)Added python import hook in multiple files to allow either
ESMForesmpy, since the name was changed in version 8.4, to allow backwards-compatibility with previous ESMpy versions.