[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

MichaelLueken · 2022-11-14T21:33:28Z

DESCRIPTION OF CHANGES:

Currently, the only machine that has a functioning GNU modulefile is Cheyenne. It has been noted that several developers don't have access to Cheyenne, so if there are issues with GNU compilers, it is difficult, if not impossible, to debug. All of the necessary libraries are available on Hera to compile the SRW App using GNU 9.2.0 with MPICH 3.3.2. A new build_hera_gnu.lua modulefile has been created so that developers who can't access Cheyenne are still able to test their changes on another machine using the GNU compiler.

The work that @danielabdi-noaa has done in PR #479 has also been merged into this PR so that fundamental tests are ready for the GNU compiler on Hera.

Updated modulefiles/build_hera_intel.lua to point to the new EPIC-maintained HPC-stack locations. Fundamental tests have been run and all successfully passed on Hera.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

TESTS CONDUCTED:

hera.gnu
The SRW App successfully compiled on Hera using GNU 9.2.0. Fundamental tests have successfully completed.
hera.intel
The SRW App successfully compiled using the new EPIC-maintained HPC-stack location for Intel. Fundamental tests have successfully completed.
fundamental test suite

ISSUE:

Completes #476

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

CONTRIBUTORS

@danielabdi-noaa - Created a fundamental test suite for Hera with GNU compiler.
@natalie-perlin - Setting up the EPIC-maintained HPC-stack locations on Hera.

venitahagerty · 2022-11-14T21:49:16Z

Machine: hera
Compiler: gnu
Job: build
Repo location: /scratch1/BMC/zrtrr/rrfs_ci/autoci/pr/1122018556/20221114213516/ufs-srweather-app
Build was Successful
If test failed, please make changes and add the following label back:
ci-hera-gnu-build

danielabdi-noaa · 2022-11-14T21:52:51Z

modulefiles/build_hera_gnu.lua

+setenv("CMAKE_C_COMPILER","mpiicc")
+setenv("CMAKE_CXX_COMPILER","mpiicpc")
+setenv("CMAKE_Fortran_COMPILER","mpiifort")


These should be the GNU couterparts i think

setenv("CMAKE_C_COMPILER","mpicc") setenv("CMAKE_CXX_COMPILER","mpicxx") setenv("CMAKE_Fortran_COMPILER","mpif90")

MichaelLueken · 2022-11-14T22:27:33Z

@danielabdi-noaa Nice catch! I have corrected this entries.

venitahagerty · 2022-11-14T22:50:43Z

Machine: hera
Compiler: gnu
Job: WE
Repo location: /scratch1/BMC/zrtrr/rrfs_ci/autoci/pr/1122018556/20221114223509/ufs-srweather-app
Build was Successful
Rocoto jobs started
Long term tracking will be done on 10 experiments
If test failed, please make changes and add the following label back:
ci-hera-gnu-WE
Experiment Succeeded on hera: community_ensemble_2mems_stoch
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: pregen_grid_orog_sfc_climo
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional
Experiment Failed on hera: MET_ensemble_verification
2022-11-14 23:52:08 +0000 :: hfe10 :: Task run_ensgridvx, jobid=37805369, in state DEAD (FAILED), ran for 33.0 seconds, exit status=256, try=1 (of 1)
All experiments completed

natalie-perlin

New hpc-stack location in EPIC-managed space. As discussed in #419, #427, ufs-community/ufs-weather-model#1465, ufs-community/ufs-weather-model#1468.
This stack in this location on Hera has been independently tested with the UFS-WM by @jkbk2004, and not changing the baseline.

modulefiles/build_hera_gnu.lua

natalie-perlin · 2022-11-15T11:40:57Z

modulefiles/build_hera_gnu.lua

+
+load("srw_common")
+
+load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.7"))


The version built with the new stack: 1.8.9 (or 1.9.1)
load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.9"))

natalie-perlin · 2022-11-15T13:02:29Z

A test with EPIC-managed stack has successfully completed on Hera. Test directory: /scratch1/NCEPDEV/nems/role.epic/sandbox/SRW/expt_dirs/test_gnu/grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16

MichaelLueken · 2022-11-15T16:17:30Z

@natalie-perlin Thank you very much for your review! I have made the suggested changes so that the Hera GNU modulefile will use the new EPIC maintained HPC-stack location.

venitahagerty · 2022-11-15T16:35:28Z

Machine: hera
Compiler: gnu
Job: WE
Repo location: /scratch1/BMC/zrtrr/rrfs_ci/autoci/pr/1122018556/20221115162014/ufs-srweather-app
Build was Successful
Rocoto jobs started
Long term tracking will be done on 10 experiments
If test failed, please make changes and add the following label back:
ci-hera-gnu-WE
Experiment Succeeded on hera: pregen_grid_orog_sfc_climo
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
Experiment Succeeded on hera: community_ensemble_2mems_stoch
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: MET_ensemble_verification
All experiments completed

danielabdi-noaa · 2022-11-15T16:50:45Z

@MichaelLueken I don't think the GNU github actions labels work on either Hera or Jet. We will need to communicate with @venitahagerty to pass compiler options for setup_WE2E_tests.sh.

./setup_WE2E_tests.sh machine account compiler test_type

In the past it used only the first two, and i don't think it has been updated to use the others.

MichaelLueken · 2022-11-15T17:01:41Z

@danielabdi-noaa Thanks for the heads up! Looking at the output from the github actions label, I see that it is being built using Intel rather than GNU.

@venitahagerty How difficult is it to update the github actions so that the test will run using:

./setup_WE2E_tests.sh machine account compiler test_type

This way, we will be able to test both the Intel and GNU compilers on Hera moving forward.

MichaelLueken · 2022-11-16T18:13:47Z

@natalie-perlin I see that the changes in ufs-weather-model PR #1468 have been merged. Would you like for me to update the Hera Intel modulefile to point to the new EPIC-maintained HPC-stack location as part of this PR as well?

MichaelLueken · 2022-11-16T18:38:20Z

On a similar note, @danielabdi-noaa, would you like for me to merge your PR #479 work into this PR? This way, all modulefile and testing modifications for Hera can be completed with this single PR.

danielabdi-noaa · 2022-11-16T18:47:55Z

@MichaelLueken Sure, please go ahead and merge them.

MichaelLueken · 2022-11-16T21:23:42Z

@natalie-perlin I have updated both the build_hera_intel.lua and build_hera_gnu.lua to use the EPIC-maintained HPC-stack locations. The fundament WE2E tests have been run for both GNU and Intel and they both successfully passed.
Please let me know if there are any issues that you see with these two files.

@danielabdi-noaa I have merged your work from PR #479 into this PR. If there are any additional changes that you would like to see, please let me know.

natalie-perlin

Thank you for the changes, approving!

MichaelLueken added the ci-hera-gnu-build Kicks off automated build test on hera with gnu label Nov 14, 2022

venitahagerty removed the ci-hera-gnu-build Kicks off automated build test on hera with gnu label Nov 14, 2022

danielabdi-noaa reviewed Nov 14, 2022

View reviewed changes

MichaelLueken force-pushed the feature/hera-gnu-modulefile branch from 671a466 to d4ceeca Compare November 14, 2022 22:26

MichaelLueken added the ci-hera-gnu-WE Kicks off automated workflow test on hera with gnu label Nov 14, 2022

MichaelLueken marked this pull request as ready for review November 14, 2022 22:27

venitahagerty removed the ci-hera-gnu-WE Kicks off automated workflow test on hera with gnu label Nov 14, 2022

danielabdi-noaa approved these changes Nov 15, 2022

View reviewed changes

natalie-perlin requested changes Nov 15, 2022

View reviewed changes

MichaelLueken force-pushed the feature/hera-gnu-modulefile branch from d4ceeca to 675f74b Compare November 15, 2022 16:14

MichaelLueken added the ci-hera-gnu-WE Kicks off automated workflow test on hera with gnu label Nov 15, 2022

venitahagerty removed the ci-hera-gnu-WE Kicks off automated workflow test on hera with gnu label Nov 15, 2022

danielabdi-noaa mentioned this pull request Nov 15, 2022

[develop] Add fundamental test suite for Hera using GNU compiler #479

Closed

38 tasks

MichaelLueken force-pushed the feature/hera-gnu-modulefile branch from 675f74b to ea7511f Compare November 16, 2022 18:56

danielabdi-noaa added 2 commits November 16, 2022 19:02

Add hera gnu fundamental test to Jenkins.

84d98a6

Remove a couple of test cases.

5df2b73

MichaelLueken force-pushed the feature/hera-gnu-modulefile branch from ea7511f to e76a718 Compare November 16, 2022 19:03

MichaelLueken changed the title ~~[develop] Add build_hera_gnu.lua modulefile.~~ [develop] Add Hera GNU modulefile and fundamental test suite. Nov 16, 2022

[develop] Add build_hera_gnu.lua file. Completes ufs-community#476.

4d01087

MichaelLueken force-pushed the feature/hera-gnu-modulefile branch from e76a718 to 4d01087 Compare November 16, 2022 21:15

MichaelLueken changed the title ~~[develop] Add Hera GNU modulefile and fundamental test suite.~~ [develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. Nov 16, 2022

MichaelLueken linked an issue Nov 16, 2022 that may be closed by this pull request

Adding a GNU modulefile for Hera #476

Closed

MichaelLueken added the run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests label Nov 16, 2022

natalie-perlin approved these changes Nov 16, 2022

View reviewed changes

MichaelLueken merged commit d03e646 into ufs-community:develop Nov 17, 2022

MichaelLueken deleted the feature/hera-gnu-modulefile branch November 17, 2022 14:27

mkavulich mentioned this pull request Feb 1, 2023

run_WE2E_tests.sh does not run the intended tests on Hera if COMPILER is not explicitly set #571

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

MichaelLueken commented Nov 14, 2022 •

edited

Loading

venitahagerty commented Nov 14, 2022

danielabdi-noaa Nov 14, 2022

MichaelLueken commented Nov 14, 2022

venitahagerty commented Nov 14, 2022 •

edited

Loading

natalie-perlin left a comment

natalie-perlin Nov 15, 2022

natalie-perlin commented Nov 15, 2022 •

edited

Loading

MichaelLueken commented Nov 15, 2022

venitahagerty commented Nov 15, 2022 •

edited

Loading

danielabdi-noaa commented Nov 15, 2022

MichaelLueken commented Nov 15, 2022

MichaelLueken commented Nov 16, 2022

MichaelLueken commented Nov 16, 2022

danielabdi-noaa commented Nov 16, 2022

MichaelLueken commented Nov 16, 2022

natalie-perlin left a comment


		load("srw_common")

		load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.7"))

[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

Conversation

MichaelLueken commented Nov 14, 2022 • edited Loading

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

ISSUE:

CHECKLIST

CONTRIBUTORS

venitahagerty commented Nov 14, 2022

danielabdi-noaa Nov 14, 2022

Choose a reason for hiding this comment

MichaelLueken commented Nov 14, 2022

venitahagerty commented Nov 14, 2022 • edited Loading

natalie-perlin left a comment

Choose a reason for hiding this comment

natalie-perlin Nov 15, 2022

Choose a reason for hiding this comment

natalie-perlin commented Nov 15, 2022 • edited Loading

MichaelLueken commented Nov 15, 2022

venitahagerty commented Nov 15, 2022 • edited Loading

danielabdi-noaa commented Nov 15, 2022

MichaelLueken commented Nov 15, 2022

MichaelLueken commented Nov 16, 2022

MichaelLueken commented Nov 16, 2022

danielabdi-noaa commented Nov 16, 2022

MichaelLueken commented Nov 16, 2022

natalie-perlin left a comment

Choose a reason for hiding this comment

MichaelLueken commented Nov 14, 2022 •

edited

Loading

venitahagerty commented Nov 14, 2022 •

edited

Loading

natalie-perlin commented Nov 15, 2022 •

edited

Loading

venitahagerty commented Nov 15, 2022 •

edited

Loading