[develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion #419

natalie-perlin · 2022-10-14T18:46:10Z

DESCRIPTION OF CHANGES:

Updated modulefiles for the following HPC systems: Cheyenne, Gaea, Hera, Orion, Jet, in particular, for the SRW 2.1 release. The changes include new paths for HPC-stack and miniconda3 installations in EPIC-managed common space. Met and Metplus verification packages were installed as part of the hpc-stack. Rocoto has also been installed on Cheyenne in common EPIC-manages space. Met and Metplus modules are not loaded explicitly as modules, but the installation paths are specified in ./ush/machine/<platform>.yaml files. The packages needed for plotting routines are included in the regional_workflow conda virtual environment.

The following types of modulefiles were updated:

./modulefiles/build_<platform>
./modulefiles/wflow<platform>
./modulefiles/tasks/<platform>/miniconda_regional_workflow
./modulefiles/tasks/<platform>/make_grid.local, make_ics.local, make_lbcs.local, make_orog.local, get_extrn_lbcs.local, get_extrn_ics.local, run_fcst.local, run_vx.local
./ush/machine/<platform>.yaml

where <platform> is Cheyenne, Jet, Hera, Orion, Gaea,
and compiler either <intel> or <gnu>; Cheyenne has both compilers

A list of the recent hpc-stack builds updated recently for the current PR (plus additional compiler-mpi combinations for Hera) could be found in the following open issue in the UFS-WM/Issues-1465:.

Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

TESTS CONDUCTED:

DEPENDENCIES:

Documentation needs to be updated, including the Chapter on making plots.

ISSUE:

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

CONTRIBUTORS (optional):

@BruceKropp-Raytheon (Jenkinks jobs building the SRW and running the tests)
@EdwardSnyder-NOAA (tests on Gaea and Hera, met/metplus verification)

Update the modulefile `build_cheyenne_gnu` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting), and rocoto

Update the modulefile `build_cheyenne_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting), and rocoto

Update the modulefile `wflow_cheyenne` and use module paths for EPIC-managed miniconda3 with the regional_workflow (running+plotting) and rocoto

Update the modulefile `build_hera_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

Update the modulefile `wflow_hera` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

Update the modulefile `build_jet_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

Update the modulefile `wflow_jet` and use module path for the EPIC-managed and updated miniconda3 with the regional_workflow (running+plotting)

Update the modulefile `build_orion_intel` and use module paths for the EPIC-managed hpc-stack and updated miniconda3 with the regional_workflow (running+plotting)

Update the modulefile `flow_orion`, use a module path for EPIC-managed miniconda3 with the regional_workflow (running+plotting)

Met and metplus installed as modules in the recent hpc-stack, update installation paths

MichaelLueken · 2022-10-17T18:06:02Z

@MichaelLueken - All the modules you mentioned, bufr/11.7.0, ncio/1.1.2, and nccmp/1.8.9.0 are built in the locations for this R-419. I just have added nccmp/1.8.9.0 today to Gaea and Orion and set as a default version (hpc-stack had different version build earlier).

@natalie-perlin - Sorry about that, I meant the paths to the hpc-stacks on Gaea, Orion, and Jet, not the updated hpc-stacks that you have created for this PR:

Orion: /work/noaa/epic-ps/hpc-stack/libs/intel/2022.1.2/modulefiles/stack
Gaea: /lustre/f2/pdata/ncep_shared/hpc-stack.epic/libs/intel/2021.3.0/modulefiles/stack
Jet: /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack.epic/libs/intel/2022.1.2/modulefiles/stack

These three stack locations would need to be updated before we can try and move forward with using them for the SRW.

natalie-perlin · 2022-10-17T20:32:08Z

@MichaelLueken - these stack locations were not built using role-epic account, but under jongkim/Jong.Kim username. There are no logs, or indication how they were configured and location of hpc-stack source code directories (only guessing), and thus only Jong Kim (@jkbk2004) could know the way they were built, and could update them.

MichaelLueken · 2022-10-17T21:01:02Z

@natalie-perlin Yes, you are correct that only Jong can update the modules in the paths that I noted above and in issue #409. This was just pointing out that we need to sync the hpc-stack used for both SRW and the weather model. It sounds like you might need to add some updated versions for the weather model to use the epic.role stack locations, while Jong would need to add several modules to his personal stack locations before the SRW would be able to use his personal stack locations. Again, the primary goal is to use a single stack for both the SRW and weather model. You have completed the necessary work to use the epic.role version for the SRW, but these locations will also need to work for the weather model, with a PR opened in the weather model repository updating the location of the stack.

panll · 2022-10-18T15:53:13Z

It was tested on Hera and works fine.

BruceKropp-Raytheon · 2022-10-18T17:45:46Z

e2e tests not yet passing on Jet.

MichaelLueken

@natalie-perlin Would it be possible to roll the compiler and mpi versions back to what is currently being used on Cheyenne, Hera, and Jet? While it is nice to see that the SRW app builds and runs the workflow using these updated compilers and mpi versions, none of our WE2E tests check for reproducibility, among other aspects.

As is, the ufs-weather-model regression tests will need to be run to see if these updated compilers and mpi versions affect results. This lies outside the scope of the ufs-srweather-app and is one of the reason why a similar PR needs to opened for the ufs-weather-model repository (the other being to ensure that both the SRW app and weather model use the same EPIC maintained stacks).

MichaelLueken · 2022-10-18T18:48:47Z

modulefiles/build_cheyenne_gnu

-module load gnu/11.2.0
+module load gnu/12.1.0


Why was the compiler version updated to the latest on Cheyenne? Would it be possible to bring this back down to gnu/11.2.0?

The attempt was to use the latest available compiler. The same libraries for the gnu/11.2.0 will be built wehn cheyenne comes back from the maintenance, if this is preferred.

Thanks @natalie-perlin. Yes, please built the same libraries for gnu/11.2.0 on Cheyenne once it is back up.

MichaelLueken · 2022-10-18T18:50:45Z

modulefiles/build_hera_intel

-module load hpc-intel/2022.1.2
-module load hpc-impi/2022.1.2
+module load hpc-intel/2022.2.0
+module load hpc-impi/2022.2.0


Why were the compiler and mpi versions updated to the latest on Hera? Would it be possible to bring these back down to intel/2022.1.2 and impi/2022.1.2?

The attempt was to use the latest available compiler. The same libraries for the intel/2021.1.2 are being built and the PR will be updates asap, if this intel/2021.1.2 is preferred (tested with the WM)

Thanks @natalie-perlin! Yes, please build the same libraries using intel/2022.1.2 on Hera.

MichaelLueken · 2022-10-18T18:52:00Z

modulefiles/build_jet_intel

-module load hpc-intel/2022.1.2
-module load hpc-impi/2022.1.2
+module load hpc-intel/2022.2.0
+module load hpc-impi/2022.2.0


Why were the compiler and mpi versions updated to the latest on Jet? Would it be possible to bring these back down to intel/2022.1.2 and impi/2022.1.2?

Sure, this is not a problem. Will prepare it for the earlier version of intel/2022.1.2. The idea was to update it to the most recent compiler/mpi combination

Thanks @natalie-perlin!

natalie-perlin · 2022-10-20T13:22:34Z

The hpc libraries and modules built with intel/2022.1.2 compiler and impi/2022.1.2 have been prepared for Hera and Jet.

reversing back to use previous compiler and impi 2022.1.2 from 2022.2.0

reversing to a verified compiler/impi version 2022.1.2 from 2022.2.0

Fixed a typo

BruceKropp-Raytheon · 2022-10-21T20:33:31Z

e2e tests verified on Jet. thanks.

natalie-perlin · 2022-11-01T18:47:04Z

Modulefiles in SRW were updated to use EPIC-managed miniconda3/4.12.0 for the workflow and running tasks. The
PR#444 (@mark-a-potts) included these changes and has been already approved and merged into develop

MichaelLueken · 2023-02-10T21:13:46Z

@natalie-perlin With the merge of PR #549 (at d036849), the ufs-srweather-app is now using your EPIC-maintained HPC-Stack locations on all machines. I am now closing this PR.

If you feel that the PR should be reopened, please let me know.

natalie-perlin added 30 commits October 14, 2022 10:38

update build_cheyenne_gnu: hpc-stack, miniconda3, rocoto

bad6666

Update the modulefile `build_cheyenne_gnu` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting), and rocoto

update build_cheyenne_intel: hpc-stack, miniconda3, rocoto

125a7de

Update the modulefile `build_cheyenne_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting), and rocoto

update wflow_cheyenne: paths for miniconda3, rocoto

335df6d

Update the modulefile `wflow_cheyenne` and use module paths for EPIC-managed miniconda3 with the regional_workflow (running+plotting) and rocoto

update build_hera_intel: hpc-stack, miniconda3

6df03f9

Update the modulefile `build_hera_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

update wflow_hera: hpc-stack, miniconda3

9d2055c

Update the modulefile `wflow_hera` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

update build_jet_intel: hpc-stack, miniconda3

869dcd8

Update the modulefile `build_jet_intel` and use module paths for EPIC-managed hpc-stack, updated miniconda3 with the regional_workflow (running+plotting)

update wflow_jet: miniconda3 path

4ce8350

Update the modulefile `wflow_jet` and use module path for the EPIC-managed and updated miniconda3 with the regional_workflow (running+plotting)

update build_orion_intel: hpc-stack, miniconda3

a61de0d

Update the modulefile `build_orion_intel` and use module paths for the EPIC-managed hpc-stack and updated miniconda3 with the regional_workflow (running+plotting)

update wflow_orion: EPIC-managed path miniconda3

bdbfffa

Update the modulefile `flow_orion`, use a module path for EPIC-managed miniconda3 with the regional_workflow (running+plotting)

Update and rename conda_regional_workflow to miniconda_regional_workflow

b128d48

Update miniconda_regional_workflow

5bc02d3

Update get_extrn_ics.local

fec443e

Update get_extrn_lbcs.local

a67c502

Update get_extrn_ics.local

79ac1cc

Update make_grid.local

a37384b

Update make_ics.local

a084d63

Update make_lbcs.local

fe23645

Update and rename make_orog.hardcoded to make_orog.local

7ea5202

Update run_fcst.local

7db26c9

Update run_vx.local

c701f03

Update miniconda_regional_workflow

bb864e8

Update run_vx.local

a5e1ca2

Update get_obs.local

3893854

Update get_extrn_ics.local

bec0ff7

Update get_extrn_lbcs.local

9677dd4

Update make_lbcs.local

aceaf50

Update run_vx.local

1cc16dd

Update gaea.yaml, met and metplus paths

c4d8d93

Met and metplus installed as modules in the recent hpc-stack, update installation paths

Update hera.yaml, met and metplus paths

40875cd

Met and metplus installed as modules in the recent hpc-stack, update installation paths

Update jet.yaml, met and metplus paths

4dbf9a2

Met and metplus installed as modules in the recent hpc-stack, update installation paths

natalie-perlin added 2 commits October 17, 2022 18:33

Update build_jet_intel

ba2b864

Update wflow_jet

643fa59

panll approved these changes Oct 18, 2022

View reviewed changes

MichaelLueken reviewed Oct 18, 2022

View reviewed changes

natalie-perlin mentioned this pull request Oct 19, 2022

Updating hpc-stack modules and miniconda locations for Hera, Gaea, Cheyenne, Orion, Jet ufs-community/ufs-weather-model#1465

Closed

Create make_orog.local

9d2094e

natalie-perlin added 3 commits October 20, 2022 11:09

Update build_jet_intel

5ad21cc

reversing back to use previous compiler and impi 2022.1.2 from 2022.2.0

Update build_hera_intel

ba56eb4

reversing to a verified compiler/impi version 2022.1.2 from 2022.2.0

Update build_jet_intel

fd22d4e

Fixed a typo

natalie-perlin mentioned this pull request Oct 25, 2022

SRW modulefiles to use updated HPC-stack libraries and miniconda3 modules, on NOAA RDHPCs, in EPIC-managed common spaces #427

Closed

danielabdi-noaa mentioned this pull request Oct 25, 2022

fix for conda on gaea #428

Closed

21 tasks

This was referenced Oct 25, 2022

Update miniconda3/4.12.0 module and regional_workflow environment on Cheyenne, Hera, Jet, Orion #431

Closed

Excessive modulefiles for local tasks #446

Closed

natalie-perlin added enhancement New feature or request and removed release This PR/issue is related to a release branch Priority: HIGH labels Nov 3, 2022

JeffBeck-NOAA changed the title ~~Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion~~ [develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion Nov 11, 2022

natalie-perlin mentioned this pull request Nov 15, 2022

[develop] Add Hera GNU modulefile and fundamental test suite and update Hera Intel modulefile. #477

Merged

15 tasks

mkavulich force-pushed the develop branch from b011cfd to 25418e8 Compare February 2, 2023 20:13

MichaelLueken closed this Feb 10, 2023

natalie-perlin deleted the develop_2 branch October 13, 2023 03:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion #419

[develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion #419

natalie-perlin commented Oct 14, 2022 •

edited

Loading

MichaelLueken commented Oct 17, 2022

natalie-perlin commented Oct 17, 2022 •

edited

Loading

MichaelLueken commented Oct 17, 2022

panll commented Oct 18, 2022

BruceKropp-Raytheon commented Oct 18, 2022

MichaelLueken left a comment

MichaelLueken Oct 18, 2022

natalie-perlin Oct 19, 2022

MichaelLueken Oct 19, 2022

MichaelLueken Oct 18, 2022

natalie-perlin Oct 19, 2022

MichaelLueken Oct 19, 2022

MichaelLueken Oct 18, 2022

natalie-perlin Oct 19, 2022

MichaelLueken Oct 19, 2022

natalie-perlin commented Oct 20, 2022

BruceKropp-Raytheon commented Oct 21, 2022

natalie-perlin commented Nov 1, 2022

MichaelLueken commented Feb 10, 2023

[develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion #419

[develop] Updated modulefiles for Cheyenne, Hera, Jet, Gaea, Orion #419

Conversation

natalie-perlin commented Oct 14, 2022 • edited Loading

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

DEPENDENCIES:

ISSUE:

CHECKLIST

LABELS (optional):

CONTRIBUTORS (optional):

MichaelLueken commented Oct 17, 2022

natalie-perlin commented Oct 17, 2022 • edited Loading

MichaelLueken commented Oct 17, 2022

panll commented Oct 18, 2022

BruceKropp-Raytheon commented Oct 18, 2022

MichaelLueken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natalie-perlin commented Oct 20, 2022

BruceKropp-Raytheon commented Oct 21, 2022

natalie-perlin commented Nov 1, 2022

MichaelLueken commented Feb 10, 2023

natalie-perlin commented Oct 14, 2022 •

edited

Loading

natalie-perlin commented Oct 17, 2022 •

edited

Loading