Skip to content

Getting Started on Stampede

Jump to bottom
Yunheng Wang edited this page Feb 23, 2021 · 11 revisions

1. Check out the code:

   git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git
   cd ufs-srweather-app

Then, check out the submodules for the SRW application.

  ./manage_externals/checkout_externals

2. Prepare to Build

2.1 Prerequisite Libraries

Visit NCEPLIBS-external wiki page for instructions to install the prerequisite libraries. You can also use the installed library in an user directory here, /work/00315/tg455890/stampede2/regional_fv3/ufs_testing2/INSTALL.

2.2 Fix files

Download the static files required for running the SRW Application from the FTP data repository or use the existing ones in an user directory here, /work/00315/tg455890/stampede2/regional_fv3/fix_am && /work/00315/tg455890/stampede2/regional_fv3/fix_orog.

2.3 Pre-staged raw initial conditions

Download raw initial conditions for your case to be run and the path should be set in config.sh as described in Section 5. A sample of GFS dataset at 2019061500 is available here, /work/00315/tg455890/stampede2/regional_fv3/gfs/2019061500.

2.4 Python modules

Load module python 3.7 and install these Python3 modules, f90nml, Jinja2, Cartopy and pygrib etc., here are the instructions to install modules f90nml and Jinja2 in your home directory,

   module load python3

   $> pip3 install --user f90nml
   $> pip3 install --user Jinja2

If you do not want to install these Python packages, you can also set up your PYTHONPATH directory to use the installed ones for Python3.7 in /home1/00315/tg455890/.local/lib/python3.7/site-packages. For example, using bash shell:

    export PYTHONPATH=/home1/00315/tg455890/.local/lib/python3.7/site-packages

3. Set up the build environment

You should load the following modules on Stampede:

module purge
module load libfabric/1.7.0
module load git/2.24.1
module load autotools/1.1
module load xalt/2.8
module load TACC

module load python3/3.7.0
module load intel/18.0.2
module load cmake/3.16.1
module load impi/18.0.2
module load pnetcdf/1.11.0
module load netcdf/4.6.2

An example of environment setting is provided here, /work/00315/tg455890/stampede2/regional_fv3/ufs_testing/build_stampede_intel.env. You can just source this file before building as (for bash shell only)

source /work/00315/tg455890/stampede2/regional_fv3/ufs_testing/build_stampede_intel.env

4. Build the executables:

   mkdir build
   cd build

Run cmake to set up the Makefile, then run make

   cmake .. -DCMAKE_INSTALL_PREFIX=..
   make -j4 |& tee build.out

If this step is successful, there should be twelve executables in ufs-srweather-app/bin including an executable for the model NEMS.exe.

5. Generate the workflow experiment:

   cd ufs-srweather-app/regional_workflow/ush
   cp config.community.sh config.sh

Edit config.sh to use an account you can charge to ACCOUNT, and the name of the experiment EXPT_SUBDIR.

MACHINE="Stampede"
ACCOUNT="your account"	
EXPT_SUBDIR="my_expt_name"

USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_SOURCE_BASEDIR_ICS="/path-to/model_data/FV3GFS"
EXTRN_MDL_FILES_ICS=( "gfs.pgrb2.0p25.f000" )
EXTRN_MDL_SOURCE_BASEDIR_LBCS="/path-to/model_data/FV3GFS"
EXTRN_MDL_FILES_LBCS=( "gfs.pgrb2.0p25.f006" "gfs.pgrb2.0p25.f012" "gfs.pgrb2.0p25.f018" "gfs.pgrb2.0p25.f024" "gfs.pgrb2.0p25.f030" "gfs.pgrb2.0p25.f036" "gfs.pgrb2.0p25.f042" "gfs.pgrb2.0p25.f048" )

Please replace /path-to/model_data/FV3GFS with the actual GFS dataset path you download in Section 2.3 above.

If you download the fix files in your own working directory, you should add this section to config.sh,

FIXgsm=/path-to/fix/fix_am
TOPO_DIR=/path-to/fix/fix_orog
SFC_CLIMO_INPUT_DIR=/path-to/fix/fix_sfc_climo

Then generate the workflow:

   ./generate_FV3SAR_wflow.sh

6. Run the workflow:

Since there is still no rocoto installed on Stampede2, users should run the SRW Application tasks one by one using the wrapper scripts in ufs-srweather-app/regional_workflow/ush/wrappers.

A help script here, /work/00315/tg455890/stampede2/regional_fv3/fv3sar.odin/run_fv3lam.sh may be useful for you to get started. Please read the README.md file in that directory. Furthermore, you can get a brief instruction about this script by runing

   $> run_fv3lam.sh -h

Basically, the script just inserts the SLURM directives (based on the settings in $EXPT_DIR/var_defns.sh) to the wrapper scripts and then run them one by one (the generated job scripts for all tasks will be in directory $EXPT_DIR/log) .

First, you run

   $> get_files.sh $EXPT_DIR/var_defns.sh

to stage the GFS initial datasets. Where $EXPT_DIR is the experimental directory created by the script generate_FV3SAR_wflow.sh in Section 5 above (Please replace $EXPT_DIR with your actual experiment path).

Then, you can run

  $> run_fv3lam.sh -m stampede  $EXPT_DIR/var_defns.sh job_name

Where the job_name is any one from [grid, orog, sfc, ics, lbc, fcst, post] in that order.

7. Plotting the output:

If you have installed "Cartopy" & "pygrib" and other required python3 modules, you can plot the SRW Application forecast by following Step 7 in Getting Started.

I still did not test the plotting scripts on Stampede and I will update this page when it is fully tested.

For more detailed information on the application, see the UFS Short-Range Weather App Users Guide.

Clone this wiki locally