Test new EPIC-maintained hpc-stack on Hera and Orion

[GeorgeGayno-NOAA]

The switch the new hpc-stack was already done on Jet. See #771.

For details on how to use the hpc-stack on Hera and Orion, see: ufs-community/ufs-weather-model#1465

The new hpc-stack is already being tested in the global-workflow: NOAA-EMC/global-workflow#1311

[GeorgeGayno-NOAA]

Unfortunately, IP v4.0.0 is not part of the stack on Hera and Orion. Will need to request that from the EPIC group.

And on Orion, I can't get the latest version of ESMF (8.4.1b07) to link properly. Temporarily using 8.3.

Will need to resolve both issues before merging.

[GeorgeGayno-NOAA]

On Orion, the link to ESMF (8.4.1b07) fails with numerous 'undefined reference' statements. Ex:

/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/esmf/8.4.1b07/lib/libesmf.a(GeomQueryTool.o): In function `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::replace(unsigned long, unsigned long, char const*, unsigned long)':
/apps/gcc-10.2.0/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:1939: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)'

[GeorgeGayno-NOAA]

Unfortunately, IP v4.0.0 is not part of the stack on Hera and Orion. Will need to request that from the EPIC group.

And on Orion, I can't get the latest version of ESMF (8.4.1b07) to link properly. Temporarily using 8.3.

Will need to resolve both issues before merging.

Updated to v4.0.0 on Orion at b19a50b.

[GeorgeGayno-NOAA]

More details on IP v4.0.0:

NOAA-EMC/global-workflow#1311 (comment)

[GeorgeGayno-NOAA]

@WalterKolczynski-NOAA - On Orion, does global_workflow care which stack UFS_UTILS points to?

• /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/modulefiles/stack or
• /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/modulefiles/stack

I can't get ESMF v8.4.1 to work using the first option (fails at the link step). Using the second option, I can use v8.4.2. I want to avoid regressing to v8.3.

[WalterKolczynski-NOAA]

@WalterKolczynski-NOAA - On Orion, does global_workflow care which stack UFS_UTILS points to?

• /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/modulefiles/stack or
• /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/modulefiles/stack

I can't get ESMF v8.4.1 to work using the first option (fails at the link step). Using the second option, I can use v8.4.2. I want to avoid regressing to v8.3.

I don't know of any as long as all the other modules we use are there. Would need to test. Of course, I don't know why NetCDF 4.9.2 requires an entirely new stack installation in the first place.

[GeorgeGayno-NOAA]

The branch at 815fc5b was compiled on Orion and Hera (using Intel) the regression tests were run. All chgres tests failed. All other tests passed. The chgres test files differed from the baseline files by what may be described as 'floating point' differences. Most of these differences were in the surface files. These differences do not indicate an error with the new stack.

[GeorgeGayno-NOAA]

The branch at 815fc5b was compiled on Orion (Intel) and Hera (Intel and GNU) and the unit tests were run locally. All tests passed.

[GeorgeGayno-NOAA]

The branch at 815fc5b was compiled on Orion (Intel) and Hera (Intel and GNU) and the unit tests were run locally. All tests passed.

Test repeated on Hera using 4496d3a. All tests passed.

[GeorgeGayno-NOAA]

The branch at 815fc5b was compiled on Orion and Hera (using Intel) the regression tests were run. All chgres tests failed. All other tests passed. The chgres test files differed from the baseline files by what may be described as 'floating point' differences. Most of these differences were in the surface files. These differences do not indicate an error with the new stack.

Test repeated on Hera using 4496d3a. Results the same as previously.