Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval #744

Merged
merged 36 commits into from
May 3, 2022
Merged

Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval #744

Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c499652
Start reorganization of static data per https://github.com/ufs-commun…
mkavulich Apr 5, 2022
bd6e44d
NOMADS wget commands now will only retry once after failure to speed …
mkavulich Apr 5, 2022
d33ad19
Address 2 main issues for cycling failures.
christinaholtNOAA Apr 19, 2022
1a08664
Source the machine file before config.sh
christinaholtNOAA Apr 19, 2022
2b99615
Adding data stores to machine files.
christinaholtNOAA Apr 19, 2022
8194cff
Make data stores configurable.
christinaholtNOAA Apr 19, 2022
4dfdd87
Ensure that the tests use only data on disk.
christinaholtNOAA Apr 19, 2022
74e3726
Revert "Source the machine file before config.sh"
mkavulich Apr 19, 2022
8f77203
Christina updates
mkavulich Apr 20, 2022
1d5b021
Remove requirement for sfc files with LBCS.
christinaholtNOAA Apr 20, 2022
3dcdde3
Fix how templates of paths are handled.
christinaholtNOAA Apr 20, 2022
6622ded
Explicitly set name of RAP/HRRR files to match disk.
christinaholtNOAA Apr 20, 2022
2b295d5
Set filenames for RAP/HRRR
christinaholtNOAA Apr 20, 2022
d6e5a80
Fix typo
mkavulich Apr 20, 2022
be8e189
Specify location of pregen data for Cheyenne
mkavulich Apr 21, 2022
6fd909e
Staged static directory
Apr 22, 2022
f6c23ee
Standarized static files following PR #744 for Odin and Stampede
ywangwof Apr 23, 2022
04c53c2
Missed one update for Jet
mkavulich Apr 25, 2022
b2d18df
Add config override for data stores on all platforms.
christinaholtNOAA Apr 26, 2022
b3b02ee
updates for gaea
mark-a-potts Apr 27, 2022
3fd79b0
Remove capture_output for WCOSS compatibility.
christinaholtNOAA Apr 27, 2022
1715e27
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 27, 2022
2b69690
Update COMIN location on Hera to match other platforms
mkavulich Apr 27, 2022
feacafe
Fix Issue #700.
christinaholtNOAA Apr 20, 2022
beb1c1c
Updated noaacloud.sh
mark-a-potts Apr 27, 2022
5265c26
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 27, 2022
0d17d8b
Updates to standardize data locations on wcoss_dell_p3
mkavulich Apr 28, 2022
c5bebe7
A few wcoss changes requested by Chan-Hoo
mkavulich Apr 28, 2022
513cf20
getting conda working correctly
mark-a-potts Apr 28, 2022
e5bf146
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 28, 2022
9a3315e
replaced srun with mpiexec due to changes in image
mark-a-potts Apr 28, 2022
0279902
Merge remote-tracking branch 'upstream/develop' into feature/standard…
mkavulich May 3, 2022
cec308a
Correct location of Orion static data
mkavulich May 3, 2022
b4a7a18
Move DOMAIN_PREGEN_BASEDIR out of NCO section since it is no longer N…
mkavulich May 3, 2022
59f0d60
Fix default location for input model files on Cheyenne
mkavulich May 3, 2022
854d3f2
Revert "Fix default location for input model files on Cheyenne"
mkavulich May 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions scripts/exregional_get_extrn_mdl_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
input_file_path=${EXTRN_MDL_SOURCE_BASEDIR_LBCS:-$EXTRN_MDL_SYSBASEDIR_LBCS}
fi

data_stores="hpss aws"
data_stores="${EXTRN_MDL_DATA_STORES}"

yyyymmddhh=${extrn_mdl_cdate:0:10}
yyyy=${yyyymmddhh:0:4}
Expand Down Expand Up @@ -141,7 +141,7 @@ if [ -n "${file_names:-}" ] ; then
fi

if [ -n "${input_file_path:-}" ] ; then
data_stores="disk hpss aws"
data_stores="disk $data_stores"
additional_flags="$additional_flags \
--input_file_path ${input_file_path}"
fi
Expand Down
22 changes: 11 additions & 11 deletions tests/WE2E/run_WE2E_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -924,6 +924,9 @@ PTMP=\"${PTMP}\""
#
if [ "${USE_USER_STAGED_EXTRN_FILES}" = "TRUE" ]; then

# Ensure we only check on disk for these files
data_stores="disk"

extrn_mdl_source_basedir=${TEST_EXTRN_MDL_SOURCE_BASEDIR:-}
if [ ! -d "${extrn_mdl_source_basedir:-}" ] ; then
print_err_msg_exit "\
Expand All @@ -934,20 +937,16 @@ machine (MACHINE):
fi
EXTRN_MDL_SOURCE_BASEDIR_ICS="${extrn_mdl_source_basedir}/${EXTRN_MDL_NAME_ICS}"
if [ "${EXTRN_MDL_NAME_ICS}" = "FV3GFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
elif [ "${EXTRN_MDL_NAME_ICS}" = "NAM" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}/for_ICS"
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}/\${yyyymmddhh}"
gsketefian marked this conversation as resolved.
Show resolved Hide resolved
else
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${yyyymmddhh}"
fi

EXTRN_MDL_SOURCE_BASEDIR_LBCS="${extrn_mdl_source_basedir}/${EXTRN_MDL_NAME_LBCS}"
if [ "${EXTRN_MDL_NAME_LBCS}" = "FV3GFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/${FV3GFS_FILE_FMT_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
elif [ "${EXTRN_MDL_NAME_LBCS}" = "GSMGFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/${FV3GFS_FILE_FMT_LBCS}/\${yyyymmddhh}"
else
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}/for_LBCS"
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${yyyymmddhh}"
fi
#
# Make sure that the forecast length is evenly divisible by the interval
Expand All @@ -968,10 +967,11 @@ boundary conditions specification interval (LBC_SPEC_INTVL_HRS):
# Locations and names of user-staged external model files for generating
# ICs and LBCs.
#
EXTRN_MDL_SOURCE_BASEDIR_ICS=${EXTRN_MDL_SOURCE_BASEDIR_ICS}
EXTRN_MDL_SOURCE_BASEDIR_ICS='${EXTRN_MDL_SOURCE_BASEDIR_ICS}'
EXTRN_MDL_FILES_ICS=( ${EXTRN_MDL_FILES_ICS[@]} )
EXTRN_MDL_SOURCE_BASEDIR_LBCS=${EXTRN_MDL_SOURCE_BASEDIR_LBCS}
EXTRN_MDL_FILES_LBCS=( ${EXTRN_MDL_FILES_LBCS[@]} )"
EXTRN_MDL_SOURCE_BASEDIR_LBCS='${EXTRN_MDL_SOURCE_BASEDIR_LBCS}'
EXTRN_MDL_FILES_LBCS=( ${EXTRN_MDL_FILES_LBCS[@]} )
EXTRN_MDL_DATA_STORES=\"$data_stores\""

fi
#
Expand Down
1 change: 1 addition & 0 deletions ...extrn_mdls_suites_community/config.grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ EXTRN_MDL_NAME_ICS="FV3GFS"
FV3GFS_FILE_FMT_ICS="grib2"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
gsketefian marked this conversation as resolved.
Show resolved Hide resolved
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
3 changes: 3 additions & 0 deletions ...mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,13 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="HRRR"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
CYCL_HRS=( "00" )

FCST_LEN_HRS="24"
LBC_SPEC_INTVL_HRS="3"

2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="HRRR"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...s_suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_GFS_v15p2"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
3 changes: 3 additions & 0 deletions ...n_mdls_suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )


DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_GFS_v15p2"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...rn_mdls_suites_nco/config.nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.get_from_HPSS_ics_HRRR_lbcs_RAP.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"

EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.subhourly_post.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.subhourly_post_ensemble_2mems.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
12 changes: 6 additions & 6 deletions ush/NOMADS_get_extrn_mdl_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ cd gfs.$yyyymmdd/$hh

#getting online analysis data
if [ $file_fmt == "grib2" ] || [ $file_fmt == "GRIB2" ]; then
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f000
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f000
else
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmanl.nemsio
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcanl.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmanl.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcanl.nemsio
fi

#getting online forecast data
Expand All @@ -60,10 +60,10 @@ echo $ifcst
echo $ifcst_str
#
if [ $file_fmt == "grib2" ] || [ $file_fmt == "GRIB2" ]; then
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f${ifcst_str}
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f${ifcst_str}
else
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmf${ifcst_str}.nemsio
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcf${ifcst_str}.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmf${ifcst_str}.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcf${ifcst_str}.nemsio
fi
#
ifcst=$[$ifcst+$nfcst_int]
Expand Down
15 changes: 12 additions & 3 deletions ush/config_defaults.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,12 +238,12 @@ EXEC_SUBDIR="bin"
# need in order to create initial and boundary condition files for a given
# cycle on the native FV3-LAM grid.
#
# FIXLAM_NCO_BASEDIR:
# DOMAIN_PREGEN_BASEDIR:
gsketefian marked this conversation as resolved.
Show resolved Hide resolved
# The base directory containing pregenerated grid, orography, and surface
# climatology files. For the pregenerated grid specified by PREDEF_GRID_NAME,
# these "fixed" files are located in:
#
# ${FIXLAM_NCO_BASEDIR}/${PREDEF_GRID_NAME}
# ${DOMAIN_PREGEN_BASEDIR}/${PREDEF_GRID_NAME}
#
# The workflow scripts will create a symlink in the experiment directory
# that will point to a subdirectory (having the name of the grid being
Expand Down Expand Up @@ -292,7 +292,8 @@ EXEC_SUBDIR="bin"
#-----------------------------------------------------------------------
#
COMIN="/path/of/directory/containing/data/files/for/IC/LBCS"
FIXLAM_NCO_BASEDIR=""
DOMAIN_PREGEN_BASEDIR=""
STMP="/base/path/of/directory/containing/model/input/and/raw/output/files"
envir="para"
NET="rrfs"
model_ver="v1.0.0"
Expand Down Expand Up @@ -711,13 +712,21 @@ EXTRN_MDL_SYSBASEDIR_LBCS=''
# EXTRN_MDL_FILES_LBCS:
# Analogous to EXTRN_MDL_FILES_ICS but for LBCs instead of ICs.
#
# EXTRN_MDL_DATA_STORES:
# A list of data stores where the scripts should look for external model
# data. The list is in priority order. If disk information is provided
# via USE_USER_STAGED_EXTRN_FILES or a known location on the platform,
# the disk location will be highest priority. Options are disk, hpss,
# aws, and nomads.
#
#-----------------------------------------------------------------------
#
USE_USER_STAGED_EXTRN_FILES="FALSE"
EXTRN_MDL_SOURCE_BASEDIR_ICS=""
EXTRN_MDL_FILES_ICS=""
EXTRN_MDL_SOURCE_BASEDIR_LBCS=""
EXTRN_MDL_FILES_LBCS=""
EXTRN_MDL_DATA_STORES=""
#
#-----------------------------------------------------------------------
#
Expand Down
25 changes: 13 additions & 12 deletions ush/machine/cheyenne.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@ QUEUE_HPSS=${QUEUE_HPSS:-"regular"}
QUEUE_FCST=${QUEUE_FCST:-"regular"}

# UFS SRW App specific paths
FIXgsm=${FIXgsm:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_am"}
FIXaer=${FIXaer:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_aer"}
FIXlut=${FIXlut:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"/glade/p/ral/jntp/UFS_CAM/fix/climo_fields_netcdf"}
FIXLAM_NCO_BASEDIR=${FIXLAM_NCO_BASEDIR:-"/needs/to/be/specified"}
staged_data_dir="/glade/p/ral/jntp/UFS_SRW_App/develop"
gsketefian marked this conversation as resolved.
Show resolved Hide resolved
FIXgsm=${FIXgsm:-"${staged_data_dir}/fix/fix_am"}
FIXaer=${FIXaer:-"${staged_data_dir}/fix/fix_aer"}
FIXlut=${FIXlut:-"${staged_data_dir}/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"${staged_data_dir}/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"${staged_data_dir}/fix/fix_sfc_climo"}
DOMAIN_PREGEN_BASEDIR=${DOMAIN_PREGEN_BASEDIR:-"${staged_data_dir}/FV3LAM_pregen"}

# Run commands for executables
RUN_CMD_SERIAL="time"
Expand All @@ -62,12 +63,12 @@ RUN_CMD_POST='mpirun -np $nprocs'
# MET/METplus-Related Paths
MET_INSTALL_DIR=${MET_INSTALL_DIR:-"/glade/p/ral/jntp/MET/MET_releases/10.0.0"}
METPLUS_PATH=${METPLUS_PATH:-"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0"}
CCPA_OBS_DIR=${CCPA_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ccpa/proc"}
MRMS_OBS_DIR=${MRMS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/mrms/proc"}
NDAS_OBS_DIR=${NDAS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ndas/proc"}
CCPA_OBS_DIR=${CCPA_OBS_DIR:-"${staged_data_dir}/obs_data/ccpa/proc"}
MRMS_OBS_DIR=${MRMS_OBS_DIR:-"${staged_data_dir}/obs_data/mrms/proc"}
NDAS_OBS_DIR=${NDAS_OBS_DIR:-"${staged_data_dir}/obs_data/ndas/proc"}
MET_BIN_EXEC=${MET_BIN_EXEC:-"bin"}

# Test Data Locations
TEST_PREGEN_BASEDIR="/glade/p/ral/jntp/UFS_SRW_app/FV3LAM_pregen"
TEST_COMIN="/glade/p/ral/jntp/UFS_SRW_app/COMGFS"
TEST_EXTRN_MDL_SOURCE_BASEDIR="/glade/p/ral/jntp/UFS_SRW_app/staged_extrn_mdl_files"
TEST_COMIN="${staged_data_dir}/COMGFS"
TEST_PREGEN_BASEDIR="${staged_data_dir}/FV3LAM_pregen"
TEST_EXTRN_MDL_SOURCE_BASEDIR="${staged_data_dir}/input_model_data"
14 changes: 7 additions & 7 deletions ush/machine/gaea.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,13 @@ QUEUE_FCST=${QUEUE_DEFAULT:-"normal"}
WTIME_MAKE_LBCS="00:60:00"

# UFS SRW App specific paths
FIXgsm=${FIXgsm:-"//lustre/f2/dev/Mark.Potts/EPIC/fix/fix_am"}
FIXaer=${FIXaer:-"/lustre/f2/dev/Mark.Potts/EPIC/fix/fix_aer"}
FIXlut=${FIXlut:-"/lustre/f2/dev/Mark.Potts/EPIC/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"/lustre/f2/dev/Mark.Potts/EPIC/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"/lustre/f2/dev/Mark.Potts/EPIC/fix/fix_sfc_climo"}
staged_data_dir="/lustre/f2/dev/Mark.Potts/EPIC/UFS_SRW_App/develop"
FIXgsm=${FIXgsm:-"${staged_data_dir}/fix/fix_am"}
FIXaer=${FIXaer:-"${staged_data_dir}/fix/fix_aer"}
FIXlut=${FIXlut:-"${staged_data_dir}/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"${staged_data_dir}/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"${staged_data_dir}/fix/fix_sfc_climo"}
TEST_EXTRN_MDL_SOURCE_BASEDIR="${staged_data_dir}/input_model_data"

RUN_CMD_SERIAL="time"
#Run Commands currently differ for GNU/openmpi
Expand All @@ -66,5 +68,3 @@ RUN_CMD_POST='srun --mpi=pmi2 -n $nprocs'
# MET Installation Locations
# MET Plus is not yet supported on gaea
# Test Data Locations
TEST_EXTRN_MDL_SOURCE_BASEDIR="/lustre/f2/dev/Mark.Potts/EPIC/SRW/model_data"

Loading