Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added support of the NSSL CCPP suite - WoFS_v0 and WE2E tests (#772)
* Added workflow for RRFS_v1nssl * Renamed FV3_RRFS_v1nssl to FV3_WoFS_v0 and added WE2E tests configuraiton files for suite FV3_WoFS_v0 * Added SUBCONUS test for WoFS suite * water_nc initial value sets to zero for FV3GFS data * Add Gaea as a supported platform for the regional_workflow (#734) * Updates to port regional workflow to gaea * Temp change with -v as batch option * new fixes for gaea/slurm * Updated time for make lbcs * added TEST data directory path * Update gaea.sh * fixes for PR * Add more parameters to CSV file containing WE2E test info (#740) ## DESCRIPTION OF CHANGES: The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests. Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description. In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file. This additional information consists of the values of the following experiment variables for each test: ``` PREDEF_GRID_NAME CCPP_PHYS_SUITE EXTRN_MDL_NAME_ICS EXTRN_MDL_NAME_LBCS DATE_FIRST_CYCL DATE_LAST_CYCL CYCL_HRS INCR_CYCL_FREQ FCST_LEN_HRS LBC_SPEC_INTVL_HRS NUM_ENS_MEMBERS ``` In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members). ## TESTS CONDUCTED: The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`. This created a new CSV file that contained the new fields (columns). The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct. ## DOCUMENTATION: The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`. This PR slightly modifies that documentation to update it. * Update directory structure of NCO mode (#743) * update vertical structure of NCO mode * update sample script for nco * Fix typo on write component of new RRFS CONUS * Default CCPP physics option is FV3_GFS_v16 (#746) * Updated the default CCPP physics option to FV3_GFS_v16 * Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local> * Adds an alternative python workflow generation path (#698) * Workflow in python starting to work. * Use new python_utils package structure. * Some bug fixes. * Use uppercase TRUE/FALSE in var_dfns * Use config.sh by default. * Minor bug fixes. * Remove config.yaml * Update to the latest develop * Remove quotes from numbers in predef grid. * Minor bug fix. * Move validity checker to the bottom of setup * Add more unit tests. * Update with python_utils changes. * Update to latest develop additions (Need to re-run regression test) * Use set_namelist and fill_jinja_template as python functions. * Replace sed regex searches with python re. * Use python realpath. * Construct settings as dictionary before passing to fill_jinja and set_namelist * Use yaml for setting predefined grid parameters. * Use xml parser for ccpp phys suite definition file. * Remove more run_command calls. * Simplify some func argument processing. * Move different config format parsers to same file. * Use os.path.join for the sake of macosx * Remove remaining func argument processing via os.environ. * Minor bug fix in set_extrn_mdl_params.sh * Add suite defn in test_data. * Minor fixes on unittest on jet. * Simplify boolean condition checks. * Include old in renaming of old directories * Fix conflicting yaml !join tag for paths and strings. * Bug fix with setting sfcperst dict. * Imitate "readlink -m" with os.path.realpath instead of os.readlink * Don't use /tmp as that is shared by multiple users. * Bug fix with cron line, maintain quotes around TRUE/FALSE. * Update to latest develop (untested) * Bug fix with existing cron line and quotes. * Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR. * Update to latest develop * More updates. * Bug fix thanks to @willmayfield! Check both starting/ending characters are brackets for shell variable to be considered an array. * Make empty EXPT_BASEDIR workable. * Update to latest develop * Update in predef grid. * Check f90nml as well. Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu> * Fix typo and crontab issue on wcoss dell in workflow python scripts (#750) * Fix typo and failure on wcoss * fix new line issue on wcoss dell * remove capture_output * Get USER from environment Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov> * Add new WE2E configs (#748) ## DESCRIPTION OF CHANGES: Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. ## TESTS CONDUCTED: Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance. ## DEPENDENCIES: None. ## DOCUMENTATION: No documentation changes are required. * Added a fixed WoF grid and the python tool to determine the write component parameters (#733) * Added a fixed WoF grid and the python tool to determine the write component parameters * Update set_predef_grid_params.sh * Renamed file as recommended and removed unused lines * Modified comment Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com> Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl> * Replace env with modulefiles in scripts (#752) * change env to mod * update we2e script * WE2E script improvements for usability (#745) ## DESCRIPTION OF CHANGES: * Modifications to `run_WE2E_tests.sh`: * Add examples to help/usage statement * Modifications to `check_expts_status.sh`: * Add arguments list that can be processed by `process_args` * Add new optional arguments: `num_log_lines`, `verbose` * Include a help/usage message ## TESTS CONDUCTED: * Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message. * Ran `check_expts_status.sh --help` from the command line and got the expected help message. * Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected. * Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message. ## DEPENDENCIES: PR #[241](ufs-community/ufs-srweather-app#241) ## DOCUMENTATION: A lot of this PR is documentation in the scripts. There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app. * Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (#744) * Bug fixes (grid size + suppress screen output from module load) (#756) ## DESCRIPTION OF CHANGES: 1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR #725 ). 2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context). ## TESTS CONDUCTED: 1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does). 2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is). * Update default SPP ISEED array in config_defaults.sh to use unique values (#759) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Modify default ISEED values for SPP * Fix grid in WE2E test * Update workflow python scripts (#760) * update python scripts * Change output file name of run_post to meet NCO standards (#758) * change output file name * change variable name * update python script * remove duplicates * add a check for empty variables * move variable to common area * clean up unnecessary comments * update scripts * remove duplicate * update python scripts * fix user-staged dir path issue in python script * Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (#763) * Add new var to we2e tests for new grids * rename we2e tests for custom grid * remove unnecessary $ * Modifications to `CODEOWNERS` file (#757) * Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them. * Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo. * Adding a python utility for summarizing compute. (#769) Adds a utility that summarizes Rocoto database computational usage information. * Add github actions for python unittests. (#747) * Add github actions for python unittests. * Include all python script in ush * Skip defining QUILTING params when it is set to False * Update py_workflow * Update unittest for set_extrn_mdl_params. * Updates from develop. Co-authored-by: Daniel Shawul <dshawul@yahoo.com> * Update sample script for NCO mode (#771) * update config.nco.sh * Add comment * Feature/noaacloud (#767) * updates for noaacloud * working version * fixes for noaacloud * added extra modules for post * removed cheyenne-specific crontab editing section (#773) * Pin down hera miniconda3 module file version. (#770) Pin down the version of miniconda3 on Hera, and do not append to the module path. * update staged data dir (#774) * Changed to smaller SUBCONUS_Ind_3km domain Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com> Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com> Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com> Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com> Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local> Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com> Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu> Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov> Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com> Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl> Co-authored-by: Michael Kavulich <kavulich@ucar.edu> Co-authored-by: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com> Co-authored-by: Daniel Shawul <dshawul@yahoo.com>
- Loading branch information
15 people
authored
Jun 9, 2022
1 parent
bc85443
commit 980d592