Update hpc stack from hpc-stack to hpc-stack-gfsv16

[emilyhcliu]

Description of the Issue
The HPC stack used in release/gfsv16 is not up-to-date for CRTM on HERA.
The updated HPC stack for gfsv16 is the hpc-stack-gfsv16 CRTM on HERA
See issues documented in global-workflow 1453

Further investigation of the HPC stacks on ORION also found that CRTM version 2.4.0 in the HPC stack is also outdated.
Issue #519 has been opened in hpc-stack repo to request updating the CRTM version 2.4.0 on all machines.
A related issue #517 was opened to update CRTM version 2.4.0 coefficients on all machines.

Based on the discussion in the GW issue #1453,
we need to do the following:

• GSI develop uses EPIC-maintain hpc-stacks and, later, spack-stack
• GSI release/gfsda.v16 uses hpc-stack-gfsv16

On HERA, updating the hpc stack to hpc-stack-gfsv16 is ready. So, I will open a PR to update the HERA modulefiles for gfsda.v16. The updated changes can be found in gfsda.v16_crtm under emilyhcliu fork.

Here are the changes

@KateFriedman-NOAA wrote a good summary about the EIB's hpc stacks maintence. It is added here for reference:

The current hpc-stack installs you're referring to are being moved away from. We will be using the following stacks moving forward:

The hpc-stack-gfsv16 for the ops GFSv16 system (dev/gfs.v16 branch). If an update is needed to this hpc-stack install to support the GFSv16 then it should be done in these installs. I believe requests for that still go to @Hang-Lei-NOAA and the https://github.com/NOAA-EMC/hpc-stack repo.
EPIC-maintained hpc-stacks for GFSv17 development system (develop branch) and then eventually spack-stack in the near future. See more info on the move to the EPIC hpc-stacks in https://github.com/NOAA-EMC/global-workflow/issues/1311 and related component issues (e.g. https://github.com/ufs-community/ufs-weather-model/issues/1465). Requests for changes to these EPC hpc-stack installs also go through https://github.com/NOAA-EMC/hpc-stack.
The current hpc-stack installs will no longer be used soon and we shouldn't spend time updating them. The GSI needs to update to use the EPIC hpc-stack intel 2022 installs on supported platforms for now. When spack-stack is ready (being tested now) we will all move to those stacks.

Note: the above is just for the R&D platforms, WCOSS2 still only has the one production hpc-stack installation and is not yet EPIC or spack-stack.

[natalie-perlin]

Two sets of hpc-stacks in EPIC-maintained space were built onOrionwith intel-2018 and intel-2022 compilers, side-by-side, to test GSI functionality.

/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2018.4
/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_gsi/

(see notes for modulefile updates in
NOAA-EMC/hpc-stack#519 (comment) )

There is an issue open #447 about issues with higher intel versions.

Building of the GSI develop branch is successful for both intel-2018, and intel-2022. Running intel-2018 regression tests still fails for 7/9 tests. Are there any updates to be done to the regression files data or location?..

22% tests passed, 7 tests failed out of 9

Total Test time (real) = 607.46 sec

The following tests FAILED:
	  1 - global_3dvar (Failed)
	  2 - global_4dvar (Failed)
	  3 - global_4denvar (Failed)
	  4 - hwrf_nmm_d2 (Failed)
	  5 - hwrf_nmm_d3 (Failed)
	  6 - rtma (Failed)
	  8 - netcdf_fv3_regional (Failed)
Errors while running CTest
Output from these tests are in: /work/noaa/epic-ps/role-epic-ps/sandbox/GSI/develop/build/Testing/Temporary/LastTest.log

[RussTreadon-NOAA]

PR #562 merged into release/gfsda.v16. Close this issue.