Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase size of RRFS CONUS grid #724

Merged
merged 8 commits into from
Apr 13, 2022
Merged

Increase size of RRFS CONUS grid #724

merged 8 commits into from
Apr 13, 2022

Conversation

BenjaminBlake-NOAA
Copy link
Collaborator

@BenjaminBlake-NOAA BenjaminBlake-NOAA commented Mar 31, 2022

DESCRIPTION OF CHANGES:

The size of the 3-km CONUS grid is increased in order to produce output on the operational HRRR / NAM nest grid without undefined points. This grid will be used by the RRFS real-time parallels, so the name should be kept as RRFS_CONUS_3km. The current RRFS_CONUS_3km grid will be kept in the workflow for any users who wish to continue using it, but it has been renamed to RRFS_CONUScompact_3km. This smaller version of the CONUS grid can be initialized off of HRRR data.

The 13-km and 25-km RRFS CONUS grids have also been updated to reflect the changes to the 3-km RRFS CONUS grid. Thanks @JeffBeck-NOAA for doing this. The current 13-km and 25-km grids have also been preserved for any users who wish to continue using them. New WE2E tests were added for the new CONUS grids.

  • List of the renamed WE2E tests (RRFS_CONUS -> RRFS_CONUScompact):
    grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
    grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
    grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
    nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR

TESTS CONDUCTED:

  • WE2E tests:
    nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km (O)
    nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR (O)
    grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 (O)
    grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional (O)
    grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp (O)
    grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 (O)
    grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR (O)
    grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp (O)
    grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2 (O)
    grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR (O)
    grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta (O)
    grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 (O)
    grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km (O)
    grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR (O)
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha (O)
    grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta (O)
    grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR (O)
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta (O)
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR (O)
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha (O)
    grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta (O)
    grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2 (O)
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR (O)
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha (O)
    grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta (O)
    grid_RRFS_NA_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1alpha (O)
    grid_RRFS_SUBCONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 (O)
    grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2 (O)
    community_ensemble_008mems (O)
    community_ensemble_2mems (O)
    deactivate_tasks (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100 (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio (O)
    get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000 (O)
    get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS (O)
    get_from_HPSS_ics_HRRR_lbcs_RAP (O)
    get_from_HPSS_ics_RAP_lbcs_RAP (O)
    inline_post (O)
    MET_ensemble_verification (O)
    MET_verification (O)
    nco_ensemble (O)
    new_ESGgrid (O)
    new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE (O)
    new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE (O)
    new_GFDLgrid (O)
    pregen_grid_orog_sfc_climo (O)
    specify_DOT_OR_USCORE (O)
    specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE (O)
    specify_RESTART_INTERVAL (O)

ISSUE:

Fixes issue mentioned in #723

CONTRIBUTORS:

@JeffBeck-NOAA @chan-hoo

@gsketefian
Copy link
Collaborator

@BenjaminBlake-NOAA Since the RRFS_CONUS_[25|13|3]km grids are currently used in many WE2E tests but are being redefined in this PR, I think it would be a good idea to rerun all such tests.

Since PR #708 by @ywangwof that is required for the release requires an update to the weather model hash, maybe it would make sense to combine the two PRs in a temporary branch so they can be tested together. If we go that route, probably a full set of tests would be the way to go. We can split the work among several people. @mkavulich Any thoughts since you've been running the full set of tests as well? @EdwardSnyder-NOAA @mark-a-potts @christopherwharrop-noaa @JeffBeck-NOAA

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 4, 2022

@gsketefian, I've submitted the following WE2E tests along with the above tests on Orion: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_NA_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1alpha
community_ensemble_2mems
nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
nco_ensemble

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 4, 2022

I found out we should update the pre-generated fix files (grid, orog, sfc_climo) of the RRFS-CONUS domains for the 'nco' mode.

@gsketefian
Copy link
Collaborator

@chan-hoo If most of the tests you ran were successful, I think we can copy the grid, orog, and sfc_climo files for the new grids to the App's data directories (e.g. on Hera, it's /scratch2/BMC/det/UFS_SRW_App/UFS_CAM/NCO_dirs/fix_lam). The old grid directories there will have to be renamed and new ones created for the new grids. And all that will have to be done on all platforms. I would do it in coordination with @mkavulich.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 5, 2022

@gsketefian @JeffBeck-NOAA @BenjaminBlake-NOAA, two tests, "grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR _suite_RRFS_v1beta", "grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha" failed at 'make_ics/make_lbcs' and 'make_ics', respectively. I suspect that there is something wrong in HRRR.

@BenjaminBlake-NOAA
Copy link
Collaborator Author

@chan-hoo Are those two tests using HRRR data for ICs/LBCs? If so then those tests will likely not work because the new versions of the RRFS CONUS grids are slightly larger than the HRRR CONUS output domain.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 5, 2022

@BenjaminBlake-NOAA, if so, should we use 'RRFS_CONUS_25km_HRRRIC' for the we2e tests using HRRR?

@BenjaminBlake-NOAA
Copy link
Collaborator Author

@chan-hoo Yes I think so. For any WE2E tests that currently use the HRRR, the new RRFS CONUS grids should not be used, and the old grids with the HRRRIC extension should be used instead.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 5, 2022

@gsketefian, I've added the fix files of the new RRFS_CONUS domains for the nco we2e tests to WCOSS_dell_p3 and Orion. The nco we2e tests except for 'nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR' were complete on the WCOSS dell and Orion successfully.

@gsketefian
Copy link
Collaborator

gsketefian commented Apr 5, 2022

@BenjaminBlake-NOAA @chan-hoo Some comments and questions:

  1. After talking with @JeffBeck-NOAA about this, we thought the "HRRIC" addition to the original grids' names is somewhat misleading because people may think they can only be initialized with HRRR. We think something like "RRFS_CONUScompact_XXkm" is more descriptive. Can we make this change?

  2. Since the RRFS_CONUScompact_3km is still of interest (at least to DTC) and the RRFS_CONUScompact_13km is of interest to CMAQ (I think according to @BenjaminBlake-NOAA), we should have WE2E tests for these that use HRRR ICs and RAP LBCs.

  3. Do the tests for the new domains now all use FV3GFS for ICs and LBCs? Or do you also use RAP? It would be nice to see in the PR message a list of all the tests that are using the new grids (these are actually all new tests) as well as a list of "new" tests that are using the original grids.

  4. We will have to update the pre-generated (grid, orog, and sfc_climo) files on all supported level-1 platforms. @chan-hoo Can you list the paths of those files on Orion? @EdwardSnyder-NOAA @mark-a-potts can you guys help transfer those files over to the other platforms (if you have access to Orion)? I can't work on that for a while since I have 2 significant PRs to review/finish soon.

Thanks.

@BenjaminBlake-NOAA
Copy link
Collaborator Author

@gsketefian Your proposed name change with 1) is fine by me. I think Jacob or Curtis suggested using HRRRIC, but I agree that it could potentially be misleading. I can go ahead and update my feature/rrfsconus branch with those name changes today. I will defer to @chan-hoo for items 2-4

@gsketefian
Copy link
Collaborator

@JacobCarley-NOAA @CurtisAlexander-NOAA I suggested changing the old (original) grid names from "RRFS_CONUS_XXkm_HRRRIC" to "RRFS_CONUScompact_XXkm" since (1) "HRRRIC" might mislead people into thinking these grids can only be initialized with HRRR, and (2) "compact" gives a better idea of how these grids relate to the new, more extensive ones. What do you think?

@EdwardSnyder-NOAA
Copy link
Collaborator

@chan-hoo - I'm getting a permission denied when trying to access these directories: ("/work/noaa/fv3-cam/UFS_SRW_App/FV3LAM_pregen/" and "/work/noaa/fv3-cam/UFS_SRW_App/FIXLAM_NCO_BASE") on Orion. But yeah, I can help out. Perhaps the new data is staged elsewhere?

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 5, 2022

@EdwardSnyder-NOAA , can you try it again? I've changed their permissions.
drwxrwsrw- 8 chjeon fv3-cam 4096 Apr 5 12:36 FIXLAM_NCO_BASE
drwxrwsrwx 14 chjeon fv3-cam 4096 Apr 5 12:37 FV3LAM_pregen

@EdwardSnyder-NOAA
Copy link
Collaborator

@chan-hoo - Sorry I didn't realize this before, but I also don't have access to the "UFS_SRW_App" directory. Getting a permission denied there too.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 5, 2022

@EdwardSnyder-NOAA, it is strange.
drwxrwSrw- 7 chjeon fv3-cam 4096 Mar 31 07:50 UFS_SRW_App
I'll copy the data to Hera once it is back tomorrow and let you know where you can find it.

@mkavulich
Copy link
Collaborator

@chan-hoo As Gerard mentioned, I am in the middle of unifying the staged data across platforms, so I can take care of copying the new pregenerated files to from Orion to Hera, Jet, and Cheyenne when the maintenance period is finished today.

Also, this is not directly related to this PR, but is there a reason to keep the pre-generated data for NCO and community cases separate? Are the output files from make_grid, make_orog, and make_sfc_climo not the same for NCO and community cases using the same grid?

In my proposal to simplify and standardize the staged data (ufs-community/ufs-srweather-app#231) I proposed merging the directories for pregenerated data, but anyone thinks they should be kept separate going forward please let me know.

@JacobCarley-NOAA
Copy link
Collaborator

@gsketefian @chan-hoo 'compact' is good with me as well.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 6, 2022

@mkavulich, I agree with you. I don't see any reasons to keep both of them separately.

@mkavulich
Copy link
Collaborator

@chan-hoo I am also having trouble accessing the files in /work/noaa/fv3-cam/UFS_SRW_App

Orion-login-1 /work/noaa/fv3-cam$ ls -al UFS_SRW_App/
ls: cannot access UFS_SRW_App/COMGFS: Permission denied
ls: cannot access UFS_SRW_App/NaturalEarth: Permission denied
ls: cannot access UFS_SRW_App/..: Permission denied
ls: cannot access UFS_SRW_App/develop: Permission denied
ls: cannot access UFS_SRW_App/FV3LAM_pregen: Permission denied
ls: cannot access UFS_SRW_App/FIXLAM_NCO_BASE: Permission denied
ls: cannot access UFS_SRW_App/.: Permission denied
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
d????????? ? ? ? ?            ? COMGFS
d????????? ? ? ? ?            ? develop
d????????? ? ? ? ?            ? FIXLAM_NCO_BASE
d????????? ? ? ? ?            ? FV3LAM_pregen
d????????? ? ? ? ?            ? NaturalEarth

I see in your ls output that the permissions are drwxrwSrw-; the capital S indicates that we do not have "execute" permissions for that directory which are needed to view the directory if the setuid bit is set. I believe chmod g+x /work/noaa/fv3-cam/UFS_SRW_App will fix this issue (might need to be done recursively).

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 6, 2022

@mkavulich, Oh.. I got it. Thank you! I've changed the permission.
drwxrwsrwx 7 chjeon fv3-cam 4096 Mar 31 07:50 UFS_SRW_App
Can you try it again?

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 6, 2022

@gsketefian, all the we2e tests were complete successfully.

@mkavulich
Copy link
Collaborator

@chan-hoo Looks good! Transferring the files to other machines now, I can let you know when that is complete.

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 6, 2022

@gsketefian @JeffBeck-NOAA, I found one we2e test failed ('grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16'). The error says: "FATAL from PE 4: compute_qs: saturation vapor pressure table overflow, nbad= 1". However, I found out this was not caused by this change (#673). I'll remove this test from my test list.

@mkavulich
Copy link
Collaborator

@chan-hoo I can confirm you can ignore that failure. Thanks for bringing it up because I forgot to open a new issue for that one, it's open now at #731

@JeffBeck-NOAA
Copy link
Collaborator

@BenjaminBlake-NOAA, do you have a location on disk somewhere for the WE2E tests run with the new grids. Do they look OK?

@BenjaminBlake-NOAA
Copy link
Collaborator Author

@JeffBeck-NOAA I believe @chan-hoo has some output on WCOSS from his WE2E tests that I could take a look at for the new 13-km and 25-km grids. @chan-hoo can you point me to that when you get a chance? Thanks

@chan-hoo
Copy link
Collaborator

chan-hoo commented Apr 6, 2022

@BenjaminBlake-NOAA @JeffBeck-NOAA, you can find the results on the dell: /gpfs/dell2/emc/modeling/noscrub/Chan-Hoo.Jeon/test_srw/expt_dirs/

@mkavulich
Copy link
Collaborator

The new pregen data is now on Hera, Jet, and Cheyenne, but I have not yet overwritten the old data. I will do so once this PR is merged.

@gsketefian
Copy link
Collaborator

@JeffBeck-NOAA The latest plots (whole grids + grid corners) are here on Hera:

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/TEST_Ligia_namelists/ufs-srweather-app/regional_workflow/ush/NCL/*Jeff_CONUS*.png

The 25km and 13km native grids look good to me, i.e. they look like they're the best you can do to match the 3km native grid.
The write-component grids are not symmetric with respect to the native grids though, and they're also not symmetric with respect to the HRRR grid (the 3km write-component grid tries to match the HRRR grid as closely as possible). We can further refine the 25km and 13km grids if you like to get them as close as possible to the HRRR grid.

@mkavulich mkavulich merged commit 05099cd into ufs-community:develop Apr 13, 2022
christinaholtNOAA added a commit to NOAA-GSL/regional_workflow that referenced this pull request Jun 7, 2022
* Add missing user-defined stochastic physics options; fix stochastic physics seed generation script (ufs-community#704)

## DESCRIPTION OF CHANGES: 
Add missing user-defined options for tendency-based stochastic physics and fix the ensemble-based seed generation script to work regardless of whether stochastic physics is turned on or not.

## TESTS CONDUCTED: 
Tested on Hera using the following WE2E configurations with and without stochastic physics:

config.grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
config.community_ensemble_2mems.sh

## ISSUE (optional): 
[Issue ufs-community#702](ufs-community#702)

## CONTRIBUTORS (optional): 
Thanks to @mkavulich and @chan-hoo for finding this problem.

* Add namelist option for netCDF4 when running with the 3-km NA domain; update NAM HPSS settings and WE2E tests (ufs-community#707)

* Change to netcdf4 when using the NA 3-km domain

* Update HPSS paths for NAM data

* Update NAM HPSS locations and dates for WE2E tests.

* Remove lines from merge.

* Tweaks to allow compiler and build_env_fn to be specified in the run_WE2E_test.sh script (ufs-community#711)

* Changed 20200304 to 20200303 in ush/mrms_pull_topofhour.py (ufs-community#712)

* Remove unused rocoto directory in ush (ufs-community#720)

* Fix bug for nco we2e tests on Orion; re-organize we2e input data and nco we2e tests (ufs-community#713)

* Update machine script for orion

* Update machine script for wcoss_dell_p3

* Update we2e run script for wcoss and orion

* Reorganize nco we2e tests

* remove machine based logic

* Add symlink for nco inline post test

* Added stand-alone verification scripts (feature/issue_683_standaloneVX) (ufs-community#726)

* Grid-stat and point-stat run scripts.

* Stand-alone scripts for verification.

* Added comments to gridvx scripts.

* Added qsub_job.sh and added comments to provide context on running Vx.

* remove machine base logic (ufs-community#727)

* Allow user-defined file names for input template files (ufs-community#717)

* Allow multiple template names

* parameterize file_TMPL_FN and add a we2e test

* Increase maxtries_task for make_grid/orog/sfc_climo

* Modify file name and description

* Changes to RRFS 3- and 13-km domains, setup.sh script bug fixes, make_ics task modification, and tweaks to stochastic physics namelist settings (ufs-community#721)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Change sotyp_from_climo to "true" based on operational RAP grib2 files.

* Update for changes to stochastic physics namelist options.

* Check for DO_ENSEMBLE="TRUE" when running ensemble verification and turn of VX when running in NCO mode.

* Revert to 3-km domain.

* Remove commented-out GFDL grid for the RRFS_NA_13km domain

* Add RRFS_NA_13km WE2E test

* Changes to comments.

* Adding 25 km tests to Jet/Hera suites. (ufs-community#718)

* Add a small 3km predefined grid over Indianapolis for testing (ufs-community#725)

* Add 3km grid over Indianapolis.  This is about 600km x 600km in extent (200 x 200 grid points).  It is intended for use in the WE2E tests.

* Edit comments.

* Use Python tool for get_extrnl_mdl_file tasks (ufs-community#681)

These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file.

The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools.

* Increase size of RRFS CONUS grid (ufs-community#724)

Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov>
Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov>

* add include-style quality mark options in metplus confs (ufs-community#738)

* Add Gaea as a supported platform for the regional_workflow (ufs-community#734)

* Updates to port regional workflow to gaea

* Temp change with -v as batch option

* new fixes for gaea/slurm

* Updated time for make lbcs

* added TEST data directory path

* Update gaea.sh

* fixes for PR

* Add more parameters to CSV file containing WE2E test info (ufs-community#740)

## DESCRIPTION OF CHANGES: 
The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests.  Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description.  In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file.  This additional information consists of the values of the following experiment variables for each test:
```
PREDEF_GRID_NAME
CCPP_PHYS_SUITE
EXTRN_MDL_NAME_ICS
EXTRN_MDL_NAME_LBCS
DATE_FIRST_CYCL
DATE_LAST_CYCL
CYCL_HRS
INCR_CYCL_FREQ
FCST_LEN_HRS
LBC_SPEC_INTVL_HRS
NUM_ENS_MEMBERS
```
In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members).  

## TESTS CONDUCTED: 
The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`.  This created a new CSV file that contained the new fields (columns).  The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct.

## DOCUMENTATION:
The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`.  This PR slightly modifies that documentation to update it.

* Update directory structure of NCO mode (ufs-community#743)

* update vertical structure of NCO mode

* update sample script for nco

* Fix typo on write component of new RRFS CONUS

* Default CCPP physics option is FV3_GFS_v16 (ufs-community#746)

* Updated the default CCPP physics option to FV3_GFS_v16

* Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh

Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>

* Adds an alternative python workflow generation path (ufs-community#698)

* Workflow in python starting to work.

* Use new python_utils package structure.

* Some bug fixes.

* Use uppercase TRUE/FALSE in var_dfns

* Use config.sh by default.

* Minor bug fixes.

* Remove config.yaml

* Update to the latest develop

* Remove quotes from numbers in predef grid.

* Minor bug fix.

* Move validity checker to the bottom of setup

* Add more unit tests.

* Update with python_utils changes.

* Update to latest develop additions (Need to re-run regression test)

* Use set_namelist and fill_jinja_template as python functions.

* Replace sed regex searches with python re.

* Use python realpath.

* Construct settings as dictionary before passing to fill_jinja and set_namelist

* Use yaml for setting predefined grid parameters.

* Use xml parser for ccpp phys suite definition file.

* Remove more run_command calls.

* Simplify some func argument processing.

* Move different config format parsers to same file.

* Use os.path.join for the sake of macosx

* Remove remaining func argument processing via os.environ.

* Minor bug fix in set_extrn_mdl_params.sh

* Add suite defn in test_data.

* Minor fixes on unittest on jet.

* Simplify boolean condition checks.

* Include old in renaming of old directories

* Fix conflicting yaml !join tag for paths and strings.

* Bug fix with setting sfcperst dict.

* Imitate "readlink -m" with os.path.realpath instead of os.readlink

* Don't use /tmp as that is shared by multiple users.

* Bug fix with cron line, maintain quotes around TRUE/FALSE.

* Update to latest develop (untested)

* Bug fix with existing cron line and quotes.

* Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR.

* Update to latest develop

* More updates.

* Bug fix thanks to @willmayfield! Check both starting/ending
characters are brackets for shell variable to be considered an array.

* Make empty EXPT_BASEDIR workable.

* Update to latest develop

* Update in predef grid.

* Check f90nml as well.

Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>

* Fix typo and crontab issue on wcoss dell in workflow python scripts (ufs-community#750)

* Fix typo and failure on wcoss

* fix new line issue on wcoss dell

* remove capture_output

* Get USER from environment

Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>

* Add new WE2E configs (ufs-community#748)

## DESCRIPTION OF CHANGES: 
Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. 

In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. 

## TESTS CONDUCTED: 
Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance.

## DEPENDENCIES:
None.

## DOCUMENTATION:
No documentation changes are required.

* Added a fixed WoF grid and the python tool to determine the write component parameters (ufs-community#733)

* Added a fixed WoF grid and the python tool to determine the write component parameters

* Update set_predef_grid_params.sh

* Renamed file as recommended and removed unused lines

* Modified comment

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>

* Replace env with modulefiles in scripts (ufs-community#752)

* change env to mod

* update we2e script

* WE2E script improvements for usability (ufs-community#745)

## DESCRIPTION OF CHANGES: 
* Modifications to `run_WE2E_tests.sh`:
  * Add examples to help/usage statement
* Modifications to `check_expts_status.sh`:
  * Add arguments list that can be processed by `process_args`
  * Add new optional arguments:  `num_log_lines`, `verbose`
  * Include a help/usage message

## TESTS CONDUCTED:
* Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message.
* Ran `check_expts_status.sh --help` from the command line and got the expected help message.
* Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected.
* Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message.
 
## DEPENDENCIES:
PR #[241](ufs-community/ufs-srweather-app#241)

## DOCUMENTATION:
A lot of this PR is documentation in the scripts.  There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app.

* Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (ufs-community#744)

* Bug fixes (grid size + suppress screen output from module load) (ufs-community#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).

* Update default SPP ISEED array in config_defaults.sh to use unique values (ufs-community#759)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Modify default ISEED values for SPP

* Fix grid in WE2E test

* Update workflow python scripts (ufs-community#760)

* update python scripts

* Change output file name of run_post to meet NCO standards (ufs-community#758)

* change output file name

* change variable name

* update python script

* remove duplicates

* add a check for empty variables

* move variable to common area

* clean up unnecessary comments

* update scripts

* remove duplicate

* update python scripts

* fix user-staged dir path issue in python script

* Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (ufs-community#763)

* Add new var to we2e tests for new grids

* rename we2e tests for custom grid

* remove unnecessary $

* Modifications to `CODEOWNERS` file (ufs-community#757)

* Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them.

* Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo.

* Adding a python utility for summarizing compute. (ufs-community#769)

Adds a utility that summarizes Rocoto database computational usage information.

* Add github actions for python unittests. (ufs-community#747)

* Add github actions for python unittests.

* Include all python script in ush

* Skip defining QUILTING params when it is set to False

* Update py_workflow

* Update unittest for set_extrn_mdl_params.

* Updates from develop.

Co-authored-by: Daniel Shawul <dshawul@yahoo.com>

* Update sample script for NCO mode (ufs-community#771)

* update config.nco.sh

* Add comment

* Feature/noaacloud (ufs-community#767)

* updates for noaacloud

* working version

* fixes for noaacloud

* added extra modules for post

* removed cheyenne-specific crontab editing section (ufs-community#773)

* Pin down hera miniconda3 module file version. (ufs-community#770)

Pin down the version of miniconda3 on Hera, and do not append to the module path.

* update staged data dir (ufs-community#774)

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com>
Co-authored-by: michelleharrold <harrold@ucar.edu>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: BenjaminBlake-NOAA <52074832+BenjaminBlake-NOAA@users.noreply.github.com>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov>
Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>
Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com>
Co-authored-by: Yunheng Wang <47898913+ywangwof@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
Co-authored-by: Daniel Shawul <dshawul@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants