Skip to content

Commit

Permalink
Allow names of external model files to be different on disk vs. from …
Browse files Browse the repository at this point in the history 
…HPSS (#221)

Allow names of external model files to be different on disk vs. from HPSS.

Motivation:
----------
At least on jet, the names of the external model files can be different depending on whether they are being copied from disk or are being extracted from an archive file in HPSS (for the same external model).  Currently, the workflow assumes that these file names are the same regardless of where/how they are being obtained.  Thus, the workflow breaks on jet.  This PR generalizes the workflow to allow for the external model file names to be different (if necessary) depending on where/how they're being obtained.

Summary of Modifications:
------------------------
* Allow the names of the external model files copied from or linked to on disk to be different than the ones extracted from an archive file in HPSS.  This is the case on jet.
* In each of the get_extrn_ics and get_extrn_lbcs workflow tasks, after obtaining the external model files (either from disk or from HPSS), create a new file and save in it certain parameters that depend on this process of obtaining the external model files (e.g. the names of the external model files that were obtained, since now these names depend on whether they are obtained from disk or HPSS).  Note that one file is created for the get_extrn_ics task and another one for the get_extrn_lbcs task.  These two files are actually shell scripts containing variable definitions that get sourced later by the make_ics and make_lbcs tasks, respectively.  This change removes the need to call the get_extrn_mdl_file_dir_info again as part of the make_ics and make_lbcs tasks.
* Modify the way the temporary fix for the suite_FV3_GFS_2017_gfdlmp_regional physics suite works such that the suite definition file (SDF) is not changed in the ufs_weather_model directory where the clone of the ufs-weather-model repo is located (so that no changes to tracked files are reported in the clone of that repo) and instead, the SDF is first copied to the experiment directory and changed there.


Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Rename local variables to be lower case.
(B) Fix/add/delete comments and/or informational and/or error messages.
(C) Remove theia stanza in machine-dependent case-statement.
(D) Remove calls to save_shell_opts and restore_shell_opts in the machine-dependent case-statement because those calls were to suppress output from the "module load" commands that were previously located in this case-statement, but the "module load"s have been moved out of this case-statement and into the ush/load_modules_run_task.sh script.
(E) Remove the call to the get_extrn_mdl_file_dir_info funtion because the information that this call returned is now obtained by sourcing new variable definition files created by the get_extrn_ics and get_extrn_lbcs workflow tasks.  The full paths to these new files (actually shell scripts that just contain the variable definitions) are, respectively:
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_ICS}/ICS/extrn_mdl_ics_var_defns.sh
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_LBCS}/LBCS/extrn_mdl_ics_var_defns.sh
(F) Remove some of the arguments to the call to scripts/exregional_make_ics.sh since those arguments are now obtained by sourcing the new variable definitions file extrn_mdl_ics_var_defns.sh created by the get_extrn_ics task.
(G) Remove some of the arguments to the call to scripts/exregional_make_lbcs.sh since those arguments are now obtained by sourcing the new variable definitions file extrn_mdl_lbcs_var_defns.sh created by the get_extrn_lbcs task.
(H) For clarity, change name of workflow variable from EXTRN_MDL_LBC_UPDATE_FHRS to EXTRN_MDL_LBC_SPEC_FHRS (where "SPEC_FHRS" stands for specification forecast hours, because the LBCs aren't being updated at these forecast hours, they're being specified).  Also, do same for the local (i.e. lowercase) version of this variable.
(I) For clarity, change name of workflow variable from LBC_UPDATE_INTVL_HRS to LBC_SPEC_INTVL_HRS for the same reason as explained above in (I).
(J) Change LBC_SPEC_INTVL_HRS from 6 to 3 to test having more than one LBC specification time after hour 0 (the forecast goes to hour 6).  For now, do this only for those WE2E tests having both EXTRN_MDL_NAME_ICS and EXTRN_MDL_NAME_LBCS set to "FV3GFS".  Don't do it for the external model being set to "GSMGFS" because that model does not provide output with 3-hour frequency, only 6-hour frequency (I think).

File-by-file modifications:
--------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES: (A), (B), (H)
* Use the function check_var_valid_vals to ensure that the value specified for ICS_OR_LBCS is valid.
* In the call to the function get_extrn_mdl_file_dir_info, instead of returning only one set of external model file names (varname_extrn_mdl_fns), return two sets -- one if the files are on disk (varname_extrn_mdl_fns_on_disk) and another if the files are in archive file on HPSS (varname_extrn_mdl_fns_in_arcv).

jobs/JREGIONAL_MAKE_ICS: (A), (B), (C), (D), (E), (F)

jobs/JREGIONAL_MAKE_LBCS: (A), (B), (C), (D), (E), (G)

jobs/JREGIONAL_RUN_POST: (B)

scripts/exregional_get_extrn_files.sh:
* Rename to scripts/exregional_get_extrn_mdl_files.sh.  See below for other changes.

scripts/exregional_get_extrn_mdl_files.sh: (A), (B), (H)
* Add new input argument ics_or_lbcs instead of assuming that an environment variable named ICS_OR_LBCS exists.
* Add new input argument extrn_mdl_cdate that will be written to the new variable definition file (extrn_mdl_ics_var_defns.sh or extrn_mdl_lbcs_var_defns.sh) that this script now creates (see bullet point below).
* Replace input argument EXTRN_MDL_FNS with the two new input arguments extrn_mdl_fns_on_disk and extrn_mdl_fns_in_arcv that are the external model file names if copying them from disk and if extracting them from an archive file(s), respectively.  Also, replace EXTRN_MDL_FNS throughout the script with either extrn_mdl_fns_on_disk or extrn_mdl_fns_in_arcv as appropriate.  This is part of the modifications to allow the names of the external model files copied from or linked to on disk to be different than the ones extracted from an archive file in HPSS (which is the case on jet).
* Remove the fix for resetting external model file names on jet because this is now done by the use of two different input arguments for the file names (extrn_mdl_fns_on_disk and extrn_mdl_fns_in_arcv).
* Add code that, after the external model files are obtained (copied, linked to, or extracted from archive), creates a new file (a shell script) that contains the definitions of several external-model related variables that will be needed by downstream tasks.  For the ICs (i.e. the get_extrn_ics task), the full path to this file is at
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_ICS}/ICS/extrn_mdl_ics_var_defns.sh
For the LBCs (i.e. the get_extrn_ics task), the full path to this file is at
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_LBCS}/LBCS/extrn_mdl_ics_var_defns.sh
These files are sourced by the make_ics and make_lbcs tasks, respectively, to obtain the necessary variable definitions.

scripts/exregional_make_ics.sh: (A), (B), (F)

scripts/exregional_make_lbcs.sh: (A), (B), (G), (H)

tests/baseline_configs/config.nco_conus.sh: (I), (J)

tests/baseline_configs/config.nco_conus_c96.sh: (I), (J)

tests/baseline_configs/config.new_JPgrid.sh: (I), (J)

tests/baseline_configs/config.regional_001.sh: (I)

tests/baseline_configs/config.regional_002.sh: (I), (J)

tests/baseline_configs/config.regional_003.sh: (I)

tests/baseline_configs/config.regional_004.sh: (I)

tests/baseline_configs/config.regional_005.sh: (I)

tests/baseline_configs/config.regional_006.sh: (I), (J)

tests/baseline_configs/config.regional_007.sh: (I), (J)

tests/baseline_configs/config.regional_008.sh: (I), (J)

tests/baseline_configs/config.regional_009.sh: (I)

tests/baseline_configs/config.regional_010.sh: (I)

tests/baseline_configs/config.regional_011.sh: (I)

tests/baseline_configs/config.regional_012.sh: (I)

tests/baseline_configs/config.regional_013.sh: (I)

tests/baseline_configs/config.regional_014.sh: (I)

tests/baseline_configs/config.regional_015.sh: (I), (J)

ush/config_defaults.sh: (B), (I)
* Add the new workflow variables EXTRN_MDL_ICS_VAR_DEFNS_FN and EXTRN_MDL_LBCS_VAR_DEFNS_FN that specify the names of the two new external-model-associated variable definitions files that the get_extrn_ics and get_extrn_lbcs now create.
* Remove extra spaces at ends of elements of array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING.

ush/generate_FV3SAR_wflow.sh: (I)
* Put in an if-statement that renames the temporary SDF (suite_FV3_GFS_2017_gfdlmp_regional.xml.tmp) created in the experiment directory by the temporay fix in ush/setup.sh (see below) for the FV3_GFS_2017_gfdlmp_regional physics suite to the actual suite file (suite_FV3_GFS_2017_gfdlmp_regional.xml).  This code needs to be removed once the SDF for this physics suite is fixed in the dtc/develop branch of the NCAR fork of the ufs-weather-model repo.

ush/get_extrn_mdl_file_dir_info.sh: (A), (C), (H)
* In the arguments list, instead of returning only one set of external model file names (varname_extrn_mdl_fns), return two sets -- one if the files are on disk (varname_extrn_mdl_fns_on_disk) and another if the files are in archive file on HPSS (varname_extrn_mdl_fns_in_arcv).
* Remove some unused code.
* Add new local variables needed to create sets of file names for both the on-disk case and the in-archive-file case.
* Allow the input argument anl_or_fcst to contain a value that is either upper case or lower case (instead of just upper case as before).
* Change code to specify both the local array variables fns_on_disk and fns_in_arcv instead of the single array variable fns.  These two new variables are (so far) the same on all machines except Jet, in which case they may be different depending on the external model considered.
* Change the way output variables are set so that each is set only if the corresponding input argument that specifies its name is not empty.  This way, if certain output variables are not needed, the user can simply not specify them in the call to this function (in which case the process_args function will simply set them to empty strings) and not obtain any errors.

ush/set_ozone_param.sh: (B)

ush/setup.sh: (A), (I)
* Change the code that is a temporary fix for the fact that the SDF for the FV3_GFS_2017_gfdlmp_regional physics suite in the dtc/develop branch of the NCAR fork of the ufs-weather-model repo is currently incorrect so that it first copies the suite defintion file to the experiment directory and makes fixes to it there (instead of fixing it directly in its original location at sorc/ufs_weather_model/...).  As part of this change:
  a) Move the creation of the experiment directory (EXPTDIR) to before the temporary code.
  b) Move the original code that calls the function set_ozone_param in the else" portion of the "if" statement that runs the temporary code.
  These two changes can be undone later after the temporary code is removed.
  • Loading branch information
@gsketefian
gsketefian authored Jun 4, 2020
1 parent a78b7d6 commit 821b68c
Show file tree
Hide file tree
Showing 30 changed files with 1,265 additions and 629 deletions.
162 changes: 80 additions & 82 deletions jobs/JREGIONAL_GET_EXTRN_MDL_FILES
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@
# (HPSS) the files generated by the external model (specified by the
# variable EXTRN_MDL_NAME) for either the initial conditions (ICs) or the
# lateral boundary conditions (LBCs). Which of these we are considering
# depends on the value of the variable ICS_OR_LBCS, which should be
# defined in the environment (when calling this script from a rocoto
# workflow, the workflow should define this variable, e.g. using rocoto's
# <envar> tag).
# depends on the value of the variable ICS_OR_LBCS, which should be defined
# in the environment (when calling this script from a rocoto workflow,
# the workflow should define this variable, e.g. using rocoto's <envar>
# tag).
#
# Note that when we refer to ICs, we are referring to not only the
# atmospheric fields at the initial time but also various surface fields
# (which are for now time-independent) as well as the 0-th forecast hour
# LBCs. Also, when we refer to LBCs, we are referring to the LBCs excluding
# the one at the 0-th hour.
# Note that when we refer to ICs, we are referring to not only the atmospheric
# fields at the initial time but also various surface fields (which are
# for now time-independent) as well as the 0-th forecast hour LBCs. Also,
# when we refer to LBCs, we are referring to the LBCs excluding the one
# at the 0-th hour.
#
#-----------------------------------------------------------------------
#
Expand All @@ -34,7 +34,7 @@
#-----------------------------------------------------------------------
#
# Source the file defining the function that will be used to set various
# external-model-related variables.
# external-model-associated variables.
#
#-----------------------------------------------------------------------
#
Expand Down Expand Up @@ -72,32 +72,32 @@ print_info_msg "
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
This is the J-job script for the task that copies/fetches to a local di-
rectory (either from disk or HPSS) the external model files from which
This is the J-job script for the task that copies/fetches to a local
directory (either from disk or HPSS) the external model files from which
initial or boundary condition files for the FV3 will be generated.
========================================================================"
#
#-----------------------------------------------------------------------
#
# Check whether output files from the specified external model (EXTRN_-
# MDL_NAME) are available on the specified cycle date and time (CDATE).
# Check whether output files from the specified external model (EXTRN_MDL_NAME)
# are available on the specified cycle date and time (CDATE).
#
#-----------------------------------------------------------------------
#
case $EXTRN_MDL_NAME in
case ${EXTRN_MDL_NAME} in

"GSMGFS")
# The transition date from the GSMGFS to the FV3GFS was 2019061212, i.e.
# this was the first official forecast with the FV3GFS. So we set the
# last CDATE for the GSMGFS to the one 6 hours before this.
CDATE_max="2019061206"
if [ "$CDATE" -gt "$CDATE_max" ]; then
cdate_max="2019061206"
if [ "$CDATE" -gt "$cdate_max" ]; then
print_err_msg_exit "\
Output from the specified external model (EXTRN_MDL_NAME) is not availa-
ble for the specified cycle date and time (CDATE) because the latter is
later than the last forecast date and time (CDATE_max) with this model:
later than the last forecast date and time (cdate_max) with this model:
EXTRN_MDL_NAME = \"${EXTRN_MDL_NAME}\"
CDATE_max = \"${CDATE_max}\"
cdate_max = \"${cdate_max}\"
CDATE = \"${CDATE}\""
fi
;;
Expand Down Expand Up @@ -136,9 +136,9 @@ earlier than the implementation date of this model:
;;

"HRRRX")
# From the HRRR home page (https://rapidrefresh.noaa.gov/hrrr), the im-
# plementation of the first version of the operational HRRR was Septem-
# ber 30, 2014.
# From the HRRR home page (https://rapidrefresh.noaa.gov/hrrr), the
# implementation of the first version of the operational HRRR was
# September 30, 2014.
CDATE_min="2014103000"
if [ "$CDATE" -lt "$CDATE_min" ]; then
print_err_msg_exit "\
Expand All @@ -155,96 +155,95 @@ esac
#
#-----------------------------------------------------------------------
#
# Set the parameter that determines whether we want to get analysis or
# forecast files. This depends on whether we want these files to gene-
# rate initial condition (IC) and surface field files or lateral bounda-
# ry condition (LBC) files for the SAR-FV3. Also, set TIME_OFFSET_HRS,
# which is the offset (in hours) between the current cycle's starting
# time and the starting time of the external model providing the LBCs.
# Check whether the environment variable ICS_OR_LBCS is set to a valid
# value. This variable specifies whether we are getting the external
# model files for the purpose of generating initial conditions (ICs) or
# lateral boundary condtions (LBCs) for the forecast model.
#
#-----------------------------------------------------------------------
#
valid_vals_ICS_OR_LBCS=( "ICS" "LBCS" )
if [ "$ICS_OR_LBCS" = "ICS" ]; then
ANL_OR_FCST="ANL"
TIME_OFFSET_HRS="0"
elif [ "$ICS_OR_LBCS" = "LBCS" ]; then
ANL_OR_FCST="FCST"
TIME_OFFSET_HRS="$EXTRN_MDL_LBCS_OFFSET_HRS"
else
valid_vals_ICS_OR_LBCS_str=$( printf "\"%s\" " "${valid_vals_ICS_OR_LBCS[@]}" )
print_err_msg_exit "\
Invalid value specified for ICS_OR_LBCS:
ICS_OR_LBCS = \"$ICS_OR_LBCS\"
Valid values are:
${valid_vals_ICS_OR_LBCS_str}"
fi
check_var_valid_value "ICS_OR_LBCS" "valid_vals_ICS_OR_LBCS"
#
#-----------------------------------------------------------------------
#
# Create the directory EXTRN_MDL_FILES_BASEDIR_ICS if it doesn't al-
# ready exist. This is the directory in which we will create a subdi-
# rectory for each cycle (i.e. for each CDATE) in which to store the
# files from the external model.
# Set the parameter anl_or_fcst that determines whether we want to get
# analysis or forecast files. This depends on whether we want these files
# to generate initial condition and surface field files or lateral boundary
# condition files. Also, set time_offset_hrs, which is the offset in
# hours between the current cycle's starting time and the starting time
# of the external model providing the LBCs.
#
#-----------------------------------------------------------------------
#
EXTRN_MDL_FILES_DIR="${CYCLE_DIR}/${EXTRN_MDL_NAME}/${ICS_OR_LBCS}"
if [ "${ICS_OR_LBCS}" = "ICS" ]; then
anl_or_fcst="ANL"
time_offset_hrs="0"
elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
anl_or_fcst="FCST"
time_offset_hrs="${EXTRN_MDL_LBCS_OFFSET_HRS}"
fi
#
#-----------------------------------------------------------------------
#
# Create the directory specific to the current forecast (whose starting
# date and time is specified in CDATE) in which to store the external
# model files. Then change location to that directory.
# Set the name of and then create the directory in which to store the
# external model files for the current cycle (if it doesn't already exist).
# Then change location to that directory.
#
#-----------------------------------------------------------------------
#
mkdir_vrfy -p "${EXTRN_MDL_FILES_DIR}"
cd_vrfy ${EXTRN_MDL_FILES_DIR} || print_err_msg_exit "\
Could not change directory to EXTRN_MDL_FILES_DIR:
EXTRN_MDL_FILES_DIR = \"${EXTRN_MDL_FILES_DIR}\""
extrn_mdl_files_dir="${CYCLE_DIR}/${EXTRN_MDL_NAME}/${ICS_OR_LBCS}"
mkdir_vrfy -p "${extrn_mdl_files_dir}"
cd_vrfy "${extrn_mdl_files_dir}"
#
#-----------------------------------------------------------------------
#
# Call the function that sets various external-model-related variables.
# See the function defintion file for the definitions of these varia-
# bles.
# Call the function that sets various external-model-associated variables.
# See the function defintion file for the definitions of these variables.
#
#-----------------------------------------------------------------------
#
get_extrn_mdl_file_dir_info \
extrn_mdl_name="${EXTRN_MDL_NAME}" \
anl_or_fcst="${ANL_OR_FCST}" \
anl_or_fcst="${anl_or_fcst}" \
cdate_FV3SAR="${CDATE}" \
time_offset_hrs="${TIME_OFFSET_HRS}" \
varname_extrn_mdl_cdate="EXTRN_MDL_CDATE" \
varname_extrn_mdl_lbc_update_fhrs="EXTRN_MDL_LBC_UPDATE_FHRS" \
varname_extrn_mdl_fns="EXTRN_MDL_FNS" \
varname_extrn_mdl_sysdir="EXTRN_MDL_SYSDIR" \
varname_extrn_mdl_arcv_fmt="EXTRN_MDL_ARCV_FMT" \
varname_extrn_mdl_arcv_fns="EXTRN_MDL_ARCV_FNS" \
varname_extrn_mdl_arcv_fps="EXTRN_MDL_ARCV_FPS" \
varname_extrn_mdl_arcvrel_dir="EXTRN_MDL_ARCVREL_DIR"
time_offset_hrs="${time_offset_hrs}" \
varname_extrn_mdl_cdate="extrn_mdl_cdate" \
varname_extrn_mdl_lbc_spec_fhrs="extrn_mdl_lbc_spec_fhrs" \
varname_extrn_mdl_fns_on_disk="extrn_mdl_fns_on_disk" \
varname_extrn_mdl_fns_in_arcv="extrn_mdl_fns_in_arcv" \
varname_extrn_mdl_sysdir="extrn_mdl_sysdir" \
varname_extrn_mdl_arcv_fmt="extrn_mdl_arcv_fmt" \
varname_extrn_mdl_arcv_fns="extrn_mdl_arcv_fns" \
varname_extrn_mdl_arcv_fps="extrn_mdl_arcv_fps" \
varname_extrn_mdl_arcvrel_dir="extrn_mdl_arcvrel_dir" || \
print_err_msg_exit "\
Call to function get_extrn_mdl_file_dir_info failed."
#
#-----------------------------------------------------------------------
#
# Call the ex-script for this J-job and pass to it the necessary varia-
# bles.
# Call the ex-script for this J-job and pass to it the necessary variables.
#
#-----------------------------------------------------------------------
#
EXTRN_MDL_FNS_str="( "$( printf "\"%s\" " "${EXTRN_MDL_FNS[@]}" )")"
EXTRN_MDL_ARCV_FNS_str="( "$( printf "\"%s\" " "${EXTRN_MDL_ARCV_FNS[@]}" )")"
EXTRN_MDL_ARCV_FPS_str="( "$( printf "\"%s\" " "${EXTRN_MDL_ARCV_FPS[@]}" )")"
extrn_mdl_lbc_spec_fhrs_str="( "$( printf "\"%s\" " "${extrn_mdl_lbc_spec_fhrs[@]}" )")"
extrn_mdl_fns_on_disk_str="( "$( printf "\"%s\" " "${extrn_mdl_fns_on_disk[@]}" )")"
extrn_mdl_fns_in_arcv_str="( "$( printf "\"%s\" " "${extrn_mdl_fns_in_arcv[@]}" )")"
extrn_mdl_arcv_fns_str="( "$( printf "\"%s\" " "${extrn_mdl_arcv_fns[@]}" )")"
extrn_mdl_arcv_fps_str="( "$( printf "\"%s\" " "${extrn_mdl_arcv_fps[@]}" )")"

$SCRIPTSDIR/exregional_get_extrn_files.sh \
EXTRN_MDL_FNS="${EXTRN_MDL_FNS_str}" \
EXTRN_MDL_SYSDIR="${EXTRN_MDL_SYSDIR}" \
EXTRN_MDL_FILES_DIR="${EXTRN_MDL_FILES_DIR}" \
EXTRN_MDL_ARCV_FNS="${EXTRN_MDL_ARCV_FNS_str}" \
EXTRN_MDL_ARCV_FPS="${EXTRN_MDL_ARCV_FPS_str}" \
EXTRN_MDL_ARCV_FMT="${EXTRN_MDL_ARCV_FMT}" \
EXTRN_MDL_ARCVREL_DIR="${EXTRN_MDL_ARCVREL_DIR}" || \
$SCRIPTSDIR/exregional_get_extrn_mdl_files.sh \
ics_or_lbcs="${ICS_OR_LBCS}" \
extrn_mdl_cdate="${extrn_mdl_cdate}" \
extrn_mdl_lbc_spec_fhrs="${extrn_mdl_lbc_spec_fhrs_str}" \
extrn_mdl_fns_on_disk="${extrn_mdl_fns_on_disk_str}" \
extrn_mdl_fns_in_arcv="${extrn_mdl_fns_in_arcv_str}" \
extrn_mdl_sysdir="${extrn_mdl_sysdir}" \
extrn_mdl_files_dir="${extrn_mdl_files_dir}" \
extrn_mdl_arcv_fmt="${extrn_mdl_arcv_fmt}" \
extrn_mdl_arcv_fns="${extrn_mdl_arcv_fns_str}" \
extrn_mdl_arcv_fps="${extrn_mdl_arcv_fps_str}" \
extrn_mdl_arcvrel_dir="${extrn_mdl_arcvrel_dir}" || \
print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
Expand All @@ -262,8 +261,7 @@ In directory: \"${scrfunc_dir}\"
#
#-----------------------------------------------------------------------
#
# Restore the shell options saved at the beginning of this script/func-
# tion.
# Restore the shell options saved at the beginning of this script/function.
#
#-----------------------------------------------------------------------
#
Expand Down
Loading

0 comments on commit 821b68c

Please sign in to comment.