-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Add a workflow for the data assimilation preprocessing tasks. #725
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
# This group contains all the tasks needed for preprocessing tasks for | ||
# RRFS DA. | ||
|
||
default_data_preproc_task: &default_preproc | ||
account: '&ACCOUNT;' | ||
attrs: | ||
cycledefs: #cycle_type# | ||
maxtries: '1' | ||
envars: &default_envars | ||
GLOBAL_VAR_DEFNS_FP: '&GLOBAL_VAR_DEFNS_FP;' | ||
USHdir: '&USHdir;' | ||
PDY: !cycstr "@Y@m@d" | ||
cyc: !cycstr "@H" | ||
subcyc: !cycstr "@M" | ||
LOGDIR: !cycstr "&LOGDIR;" | ||
CYCLE_TYPE: '#cycle_type#' | ||
native: '{{ platform.SCHED_NATIVE_CMD }}' | ||
nodes: '{{ nnodes }}:ppn={{ ppn }}' | ||
nnodes: 1 | ||
nodesize: "&NCORES_PER_NODE;" | ||
ppn: 1 | ||
partition: '{% if platform.get("PARTITION_DEFAULT") %}&PARTITION_DEFAULT;{% else %}None{% endif %}' | ||
queue: '&QUEUE_DEFAULT;' | ||
walltime: 00:25:00 | ||
|
||
metatask_process_obs_cycle_type: | ||
var: | ||
cycledefs_type: prodcyc,prodcyc_long | ||
cycle_type: prod | ||
|
||
task_process_radarref_#cycle_type#: | ||
<<: *default_preproc | ||
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_RADARREF"' | ||
ppn: 24 | ||
dependency: | ||
or: | ||
taskdep: | ||
attrs: | ||
task: get_da_obs_radar | ||
and: | ||
not: | ||
taskvalid: | ||
attrs: | ||
task: get_da_obs_radar | ||
streq: | ||
left: do_real_time | ||
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}' | ||
timedep: '<cyclestr offset="&START_TIME_NSSLMOSAIC;">@Y@m@d@H@M00</cyclestr>' | ||
|
||
|
||
task_process_lightning_#cycle_type#: | ||
<<: *default_preproc | ||
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_LIGHTNING"' | ||
dependency: | ||
or: | ||
taskdep: | ||
attrs: | ||
task: get_da_obs_lightning | ||
and: | ||
not: | ||
taskvalid: | ||
attrs: | ||
task: get_da_obs_lightning | ||
streq: | ||
left: do_real_time | ||
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}' | ||
timedep: '<cyclestr offset="&START_TIME_LIGHTNING;">@Y@m@d@H@M00</cyclestr>' | ||
|
||
|
||
task_process_bufrobs_#cycle_type#: | ||
<<: *default_preproc | ||
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_BUFROBS"' | ||
join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;' | ||
dependency: | ||
or: | ||
taskdep: | ||
attrs: | ||
task: get_da_obs_bufr | ||
and: | ||
not: | ||
taskvalid: | ||
attrs: | ||
task: get_da_obs_bufr | ||
streq: | ||
left: do_real_time | ||
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}' | ||
timedep: '<cyclestr offset="&START_TIME_CONVENTIONAL;">@Y@m@d@H@M00</cyclestr>' | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
metadata: | ||
description: |- | ||
Tests the data retrieval and data preprocessing tasks that are | ||
precursors to running any data assimialtion components. Datasets | ||
include bufr, lightning, and radar reflectivity. | ||
|
||
workflow: | ||
PREDEF_GRID_NAME: RRFS_CONUS_25km | ||
DATE_FIRST_CYCL: '2019061500' | ||
DATE_LAST_CYCL: '2019061500' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll need some feedback from @EdwardSnyder-NOAA on what data has been staged here before this information can be nailed down. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I placed the predefined grid data on Hera, Jet, and Orion for the following grids: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @EdwardSnyder-NOAA I'm sorry for the delay on this one. I think we should pose this question at a release meeting. More generally "is there an interesting case(s) that we should use for the release?" |
||
|
||
rocoto: | ||
tasks: | ||
taskgroups: '{{ ["parm/wflow/da_data_preproc.yaml"]|include }}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These names are placeholders for what comes of the retrieve data efforts underway by @mkavulich.
I am leaving the option here to either have this task run, or to have the script know where to look on disk for the sake of efficiency in the real-time runs.