Skip to content

Run setup scripts to generate experiment

Jump to bottom
Kate Friedman edited this page Oct 25, 2022 · 29 revisions

To run the setup scripts, you need to make sure to have a copy of python3 with numpy available. The easiest way to guarantee this is to load python from the official hpc-stack installation for the machine you are on:

Hera:

module use -a /contrib/anaconda/modulefiles
module load anaconda/anaconda3-5.3.1

Orion:

module load python/3.7.5

WCOSS2:

module load python/3.8.6

S4:

module load miniconda/3.8-s4

If running with Rocoto make sure to have a Rocoto module loaded before running setup scripts:

Hera:

module load rocoto/1.3.3

Orion:

module load contrib
module load rocoto/1.3.3

WCOSS2:

module use /apps/ops/test/nco/modulefiles/
module load core/rocoto/1.3.5

S4:

module load rocoto/1.3.4

Free-forecast experiment

Scripts that will be used:

  • workflow/setup_expt.py
  • workflow/setup_xml.py

1) Run experiment generator script (creates EXPDIR and COMROT)

NOTE: The following command examples include variables for reference but users should not use environmental variables to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts.

cd workflow
./setup_expt.py forecast-only --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC] [--resdet $RESDET]
  [--pslot $PSLOT] [--configdir $CONFIGDIR] [--comrot $COMROT] [--expdir $EXPDIR] [--icsdir $ICSDIR]

...where:

  • forecast-only is the first positional argument that instructs the setup script to produce an experiment directory for forecast only experiments.
  • $APP is the target application, one of:
    • ATM: atmosphere-only [default]
    • ATMW: atm-wave
    • ATMA: atm-aerosols
    • S2S: atm-ocean-ice
    • S2SW: atm-ocean-ice-wave
    • S2SWA: atm-ocean-ice-wave-aerosols
  • $START is the start type (warm or cold [default])
  • $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
  • $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
  • $PSLOT is the name of your experiment [default: test]
  • $CONFIGDIR is the path to the /config folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/]
  • $RESDET is the FV3 resolution (i.e. 768 for C768) [default: 384]
  • $GFS_CYC is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)
  • $COMROT is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME]
  • $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME]
  • $ICSDIR is the path to the initial conditions. This is handled differently depending on whether $APP is S2S or not.
    • If $APP is ATM or ATMW, this setting is currently ignored
    • If $APP is S2S or S2SW, ICs are copied from the central location to this location and the argument is required

Examples:

Atm-only

cd workflow
./setup_expt.py forecast-only --pslot test --idate 2020010100 --edate 2020010118 --resdet 384 --gfs_cyc 4 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir

Coupled

cd workflow
./setup_expt.py forecast-only --app S2SW --pslot coupled_test --idate 2013040100 --edate 2013040100 --resdet 384 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --icsdir /some_large_disk_area/Joe.Schmo/icsdir

Coupled with aerosols

cd workflow
./setup_expt.py forecast-only --app S2SWA --pslot coupled_test --idate 2013040100 --edate 2013040100 --resdet 384 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --icsdir /some_large_disk_area/Joe.Schmo/icsdir

2) Set user and experiment settings

Go to your EXPDIR and check/change the following variables within your config.base now before running the next script:

  • ACCOUNT
  • HOMEDIR
  • STMP
  • PTMP
  • ARCDIR (location on disk for online archive used by verification system)
  • HPSSARCH (YES turns on archival)
  • HPSS_PROJECT (project on HPSS if archiving)
  • ATARDIR (location on HPSS if archiving)

If you are using cycling, also change these:

  • imp_physics from 8 (Thompson) to 11 (GFDL)
  • CCPP_SUITE to FV3_GFS_v16 (or another suite that uses GFDL) (This is a temporary measure until cycling mode works with Thompson.)

Some of those variables will be found within a machine-specific if-block so make sure to change the correct ones for the machine you'll be running on.

Now is also the time to change any other variables/settings you wish to change in config.base or other configs. Do that now. Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script.

See the following section of this wiki for more information on configuring your run:

https://github.com/NOAA-EMC/global-workflow/wiki/Configure-your-run

3) Run workflow generator script (creates ROCOTO xml in EXPDIR)

./setup_xml.py $EXPDIR/$PSLOT

Example:

./setup_xml.py /some_safe_disk_area/Joe.Schmo/expdir/test

4) Resulting files from setup scripts

You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this: load a rocoto module and then rerun setup_xml.py script again. Cron is handled differently on WCOSS-Dell so follow different instructions for setting up your rocoto cron on Mars/Venus.

Cycled experiment

Scripts that will be used:

  • workflow/setup_expt.py
  • workflow/setup_xml.py

1) Run experiment generator script (creates EXPDIR and COMROT)

NOTE: The following command examples include variables for reference but users should not use environmental variables to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts.



cd workflow
./setup_expt.py cycled --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC]
  [--resdet $RESDET] [--resens $RESENS] [--nens $NENS] [--cdump $CDUMP]
  [--pslot $PSLOT] [--configdir $CONFIGDIR] [--comrot $COMROT] [--expdir $EXPDIR] [--icsdir $ICSDIR]

(Coupling cycled not yet supported)

...where:

  • cycled is the first positional argument that instructs the setup script to produce an experiment directory for cycled experiments.
  • $APP is the target application, one of:
    • ATM: atmosphere-only [default]
    • ATMW: atm-wave
  • $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
  • $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
  • $START is the start type (warm or cold [default])
  • $GFS_CYC is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)
  • $RESDET is the FV3 resolution of the deterministic forecast [default: 384]
  • $RESENS is the FV3 resolution of the ensemble (EnKF) forecast [default: 192]
  • $NENS is the number of ensemble members [default: 20]
  • $CDUMP is the starting phase [default: gdas]
  • $PSLOT is the name of your experiment [default: test]
  • $CONFIGDIR is the path to the /config folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/]
  • $COMROT is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME]
  • $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME]
  • $ICSDIR is the path to the ICs for your run if generated separately. [default: None]

Example:

cd workflow
./setup_expt.py cycled --pslot test --configdir /home/Joe.Schmo/git/global-workflow/parm/config --idate 2020010100 --edate 2020010118 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --resdet 384 --resens 192 --nens 80 --gfs_cyc 4

Example setup_expt.py on WCOSS_C:

SURGE-slogin1 > ./setup_expt.py cycled --pslot fv3demo --idate 2017073118 --edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs

SDATE = 2017-07-31 18:00:00
EDATE = 2017-08-01 06:00:00

EDITED:  /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input.
DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only.
Please verify and delete the default file before proceeding.

SURGE-slogin1 >

The message about the config.base.default is telling you that you are free to delete it if you wish but it’s not necessary to remove. Your resulting config.base was generated from config.base.default and the default one is there for your information.

What happens if I run setup_expt.py again for an experiment that already exists:

SURGE-slogin1 > ./setup_expt.py forecast-only --pslot fv3demo --idate 2017073118 
--edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs

COMROT already exists in /gpfs/hps2/ptmp/Joe.Schmo/fv3demo

Do you wish to over-write COMROT [y/N]: y

EXPDIR already exists in /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo

Do you wish to over-write EXPDIR [y/N]: y

SDATE = 2017-07-31 18:00:00
EDATE = 2017-08-01 06:00:00

EDITED:  /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input.
DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only.
Please verify and delete the default file before proceeding.

Your COMROT and EXPDIR will be deleted and remade. Be careful with this!

2) Set user and experiment settings

Go to your EXPDIR and check/change the following variables within your config.base now before running the next script:

  • ACCOUNT
  • HOMEDIR
  • STMP
  • PTMP
  • ARCDIR (location on disk for online archive used by verification system)
  • HPSSARCH (YES turns on archival)
  • HPSS_PROJECT (project on HPSS if archiving)
  • ATARDIR (location on HPSS if archiving)

Some of those variables will be found within a machine-specific if-block so make sure to change the correct ones for the machine you'll be running on.

Now is also the time to change any other variables/settings you wish to change in config.base or other configs. Do that now. Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script.

See the following section of this wiki for more information on configuring your run:

https://github.com/NOAA-EMC/global-workflow/wiki/Configure-your-run

3) Run workflow generator script (creates ROCOTO xml in EXPDIR)

./setup_xml.py $EXPDIR/$PSLOT

Example:

./setup_xml.py /some_safe_disk_area/Joe.Schmo/expdir/test

4) Resulting files from setup scripts

You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this: load a rocoto module and then rerun setup_xml.py script again. Cron is handled differently on WCOSS-Dell so follow different instructions for setting up your rocoto cron on Mars/Venus.

Users running GFSv16

Clone this wiki locally