Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert archive jobs to proper j-jobs #1115

Merged
merged 24 commits into from
Nov 22, 2022
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
77e9aeb
GFS archive jobs designed to increase GFS workflow stability.
lgannoaa Oct 7, 2022
5dfb5e0
Remove ecflow components
lgannoaa Oct 7, 2022
0a40e5f
Merge branch 'develop' of https://github.com/NOAA-EMC/global-workflow…
lgannoaa Oct 18, 2022
f80ee72
Init commit
lgannoaa Oct 20, 2022
838ecd7
Add J-Job and ex script for archive job development
lgannoaa Nov 1, 2022
25846f7
pull Merge upstream
lgannoaa Nov 1, 2022
e729e01
Merge branch 'develop' of https://github.com/NOAA-EMC/global-workflow…
lgannoaa Nov 2, 2022
7a6b62d
Enable GDAS and GFS archive. Jobs tested on Cactus with cycled C192/C96.
lgannoaa Nov 4, 2022
103b6a2
Create a version of ecflow archive jobs in rocoto workflow
lgannoaa Nov 9, 2022
c29bca3
Code clean up for PR preparation
lgannoaa Nov 9, 2022
9fc35a9
Restore a missing file from develop
lgannoaa Nov 9, 2022
bfe11c6
Create a version that only have jjob and ex-script updates
lgannoaa Nov 10, 2022
9930520
Revert local change on vrfy script
lgannoaa Nov 10, 2022
8c045ce
Revert local fix to build issue
lgannoaa Nov 10, 2022
ed8bd67
JENKFGDAS_ARCHIVE apply filter
lgannoaa Nov 10, 2022
9715c78
Rearrange some script statement between jjob and ex-script
lgannoaa Nov 14, 2022
b97d453
Remove extra statement in JENKFGDAS_ARCHIVE
lgannoaa Nov 14, 2022
686a63d
Putting braces around variable
lgannoaa Nov 14, 2022
e802183
remove date statement from jjobs
lgannoaa Nov 14, 2022
d286dce
Apply shellcheck 2004,2086,2166,2248,2250
lgannoaa Nov 16, 2022
9d54a70
Apply shellcheck 2004,2086,2166,2248,2250 to rocoto jobs
lgannoaa Nov 16, 2022
de26d30
Apply shell check sc2004 to exgdas_enkf_earc.sh and exglobal_archive.sh
lgannoaa Nov 16, 2022
11b1718
Update syntex
lgannoaa Nov 16, 2022
823f5f6
If statement update for exgdas_enkf_earc
lgannoaa Nov 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions jobs/JENKFGDAS_ARCHIVE
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#! /usr/bin/env bash

source "$HOMEgfs/ush/preamble.sh"

#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base earc"
config_path=${EXPDIR:-${NWROOT}/gfs.${gfs_ver}/parm/config}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
config_path=${EXPDIR:-${NWROOT}/gfs.${gfs_ver}/parm/config}
config_path=${EXPDIR:-${PACKAGEROOT}/gfs.${gfs_ver}/parm/config}

for config in ${configs}; do
. ${config_path}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
done

##########################################
# Source machine runtime environment
##########################################
. ${HOMEgfs}/env/${machine}.env earc
status=$?
[[ ${status} -ne 0 ]] && exit $status

##############################################
# Obtain unique process id (pid) and make temp directory
##############################################
export pid=${pid:-$$}
export DATA=${DATA:-${DATAROOT}/${jobid:?}}
mkdir -p $DATA
cd $DATA


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


##############################################
# Determine Job Output Name on System
##############################################
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


##############################################
# Set variables used in the script
##############################################
export CDATE=${CDATE:-${PDY}${cyc}}
export CDUMP=${CDUMP:-${RUN:-"gfs"}}
aerorahul marked this conversation as resolved.
Show resolved Hide resolved
export COMPONENT=${COMPONENT:-atmos}
export n=$((ENSGRP))
aerorahul marked this conversation as resolved.
Show resolved Hide resolved

##############################################
# Begin JOB SPECIFIC work
##############################################

# ICS are restarts and always lag INC by $assim_freq hours.
export EARCINC_CYC=${ARCH_CYC}
export EARCICS_CYC=$((ARCH_CYC-assim_freq))
if [ ${EARCICS_CYC} -lt 0 ]; then
export EARCICS_CYC=$((EARCICS_CYC+24))
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move these to exgdas_enkf_earc.sh and drop the export


# EnKF update in GFS, GDAS or both
export CDUMP_ENKF=$(echo ${EUPD_CYC:-"gdas"} | tr a-z A-Z)

export ARCH_LIST="${ROTDIR}/enkf${CDUMP}.${PDY}/${cyc}/${COMPONENT}/earc${ENSGRP}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This belongs to exgdas_enkf_earc.sh and is there.


###############################################################
# Run archive script
###############################################################

$SCRgfs/exgdas_enkf_earc.sh
status=$?
[[ $status -ne 0 ]] && exit $status

###############################################################

echo "ENDED NORMALLY."

##########################################
# Remove the Temporary working directory
##########################################
cd $DATAROOT
[[ ${KEEPDATA} = "NO" ]] && rm -rf $DATA

date
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved
exit 0
105 changes: 105 additions & 0 deletions jobs/JGLOBAL_ARCHIVE
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#! /usr/bin/env bash

source "$HOMEgfs/ush/preamble.sh"

export RUN_ENVIR=${RUN_ENVIR:-"nco"}

#############################################
# Source relevant config files
#############################################
export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
configs="base arch"
config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
for config in $configs; do
. $config_path/config.$config
status=$?
[[ $status -ne 0 ]] && exit $status
done

##########################################
# Source machine runtime environment
##########################################
. $HOMEgfs/env/${machine}.env arch
status=$?
[[ $status -ne 0 ]] && exit $status

#############################################
# Source workflow level common utilities
#############################################
#### This utility is under development as of 20221020
#### will not use it at this time
#### source "${HOMEgfs}/ush/file_utils.sh"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please delete.
These are being sourced and used in exglobal_archive.sh


##############################################
# Obtain unique process id (pid) and make temp directory
##############################################
export pid=${pid:-$$}
export DATA=${DATA:-${DATAROOT}/${jobid:?}}
mkdir -p $DATA
cd $DATA


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


##############################################
# Determine Job Output Name on System
##############################################
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


##############################################
# Set variables used in the script
##############################################
export CDATE=${CDATE:-${PDY}${cyc}}
export CDUMP=${CDUMP:-${RUN:-"gfs"}}
export COMPONENT=${COMPONENT:-atmos}


##############################################
# Begin JOB SPECIFIC work
##############################################

# ICS are restarts and always lag INC by $assim_freq hours
export ARCHINC_CYC=$ARCH_CYC
export ARCHICS_CYC=$((ARCH_CYC-assim_freq))
if [ $ARCHICS_CYC -lt 0 ]; then
ARCHICS_CYC=$((ARCHICS_CYC+24))
fi

# CURRENT CYCLE
export APREFIX="${CDUMP}.t${cyc}z."
export ASUFFIX=${ASUFFIX:-$SUFFIX}

if [ $ASUFFIX = ".nc" ]; then
format="netcdf"
else
format="nemsio"
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move these to exglobal_archive.sh and drop the export


###############################################################
# Run archive script
###############################################################

${GLOBALARCHIVESH:-$SCRgfs/exglobal_archive.sh}
status=$?
[[ $status -ne 0 ]] && exit $status

###############################################################

echo "ENDED NORMALLY."

##########################################
# Remove the Temporary working directory
##########################################
cd $DATAROOT
[[ $KEEPDATA = "NO" ]] && rm -rf $DATA

date
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved
exit 0
Loading