-
Notifications
You must be signed in to change notification settings - Fork 0
Getting Started on Stampede
git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git
cd ufs-srweather-app
Then, check out the submodules for the SRW application.
./manage_externals/checkout_externals
Visit NCEPLIBS-external wiki page for instructions to install the prerequisite libraries. You can also use the installed library in an user directory here, /work/00315/tg455890/stampede2/regional_fv3/ufs_testing2/INSTALL.
Download the static files required for running the SRW Application from the FTP data repository or use the existing ones in an user directory here, /work/00315/tg455890/stampede2/regional_fv3/fix_am && /work/00315/tg455890/stampede2/regional_fv3/fix_orog.
Download raw initial conditions for your case to be run and the path should be set in config.sh
as described in Section 5. A sample of GFS dataset at 2019061500 is available here, /work/00315/tg455890/stampede2/regional_fv3/gfs/2019061500.
Load module python 3.7 and install these Python3 modules, f90nml
, Jinja2
, Cartopy
and pygrib
etc., here are the instructions to install modules f90nml
and Jinja2
in your home directory,
module load python3
$> pip3 install --user f90nml
$> pip3 install --user Jinja2
If you do not want to install these Python packages, you can also set up your PYTHONPATH
directory to use the installed ones for Python3.7 in /home1/00315/tg455890/.local/lib/python3.7/site-packages
. For example, using bash shell:
export PYTHONPATH=/home1/00315/tg455890/.local/lib/python3.7/site-packages
You should load the following modules on Stampede:
module purge
module load libfabric/1.7.0
module load git/2.24.1
module load autotools/1.1
module load xalt/2.8
module load TACC
module load python3/3.7.0
module load intel/18.0.2
module load cmake/3.16.1
module load impi/18.0.2
module load pnetcdf/1.11.0
module load netcdf/4.6.2
An example of environment setting is provided here, /work/00315/tg455890/stampede2/regional_fv3/ufs_testing/build_stampede_intel.env. You can just source this file before building as (for bash shell only)
source /work/00315/tg455890/stampede2/regional_fv3/ufs_testing/build_stampede_intel.env
mkdir build
cd build
Run cmake to set up the Makefile, then run make
cmake .. -DCMAKE_INSTALL_PREFIX=..
make -j4 |& tee build.out
If this step is successful, there should be twelve executables in ufs-srweather-app/bin
including an executable for the model NEMS.exe.
cd ufs-srweather-app/regional_workflow/ush
cp config.community.sh config.sh
Edit config.sh
to use an account you can charge to ACCOUNT
, and the name of the experiment EXPT_SUBDIR
.
MACHINE="Stampede"
ACCOUNT="your account"
EXPT_SUBDIR="my_expt_name"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_SOURCE_BASEDIR_ICS="/path-to/model_data/FV3GFS"
EXTRN_MDL_FILES_ICS=( "gfs.pgrb2.0p25.f000" )
EXTRN_MDL_SOURCE_BASEDIR_LBCS="/path-to/model_data/FV3GFS"
EXTRN_MDL_FILES_LBCS=( "gfs.pgrb2.0p25.f006" "gfs.pgrb2.0p25.f012" "gfs.pgrb2.0p25.f018" "gfs.pgrb2.0p25.f024" "gfs.pgrb2.0p25.f030" "gfs.pgrb2.0p25.f036" "gfs.pgrb2.0p25.f042" "gfs.pgrb2.0p25.f048" )
Please replace /path-to/model_data/FV3GFS
with the actual GFS dataset path you download in Section 2.3 above.
If you download the fix files in your own working directory, you should add this section to config.sh
,
FIXgsm=/path-to/fix/fix_am
TOPO_DIR=/path-to/fix/fix_orog
SFC_CLIMO_INPUT_DIR=/path-to/fix/fix_sfc_climo
Then generate the workflow:
./generate_FV3SAR_wflow.sh
Since there is still no rocoto
installed on Stampede2, users should run the SRW Application tasks one by one using the wrapper scripts in ufs-srweather-app/regional_workflow/ush/wrappers.
A help script here, /work/00315/tg455890/stampede2/regional_fv3/fv3sar.odin/run_fv3lam.sh may be useful for you to get started. Please read the README.md
file in that directory. Furthermore, you can get a brief instruction about this script by runing
$> run_fv3lam.sh -h
Basically, the script just inserts the SLURM directives (based on the settings in $EXPT_DIR/var_defns.sh
) to the wrapper scripts and then run them one by one (the generated job scripts for all tasks will be in directory $EXPT_DIR/log
) .
First, you run
$> get_files.sh $EXPT_DIR/var_defns.sh
to stage the GFS initial datasets. Where $EXPT_DIR
is the experimental directory created by the script generate_FV3SAR_wflow.sh in Section 5 above (Please replace $EXPT_DIR with your actual experiment path).
Then, you can run
$> run_fv3lam.sh -m stampede $EXPT_DIR/var_defns.sh job_name
Where the job_name
is any one from [grid, orog, sfc, ics, lbc, fcst, post]
in that order.
If you have installed "Cartopy" & "pygrib" and other required python3 modules, you can plot the SRW Application forecast by following Step 7 in Getting Started.
I still did not test the plotting scripts on Stampede and I will update this page when it is fully tested.
For more detailed information on the application, see the UFS Short-Range Weather App Users Guide.