Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[develop] Add a workflow for the data assimilation preprocessing tasks. #725

Closed
wants to merge 4 commits into from
Closed

[develop] Add a workflow for the data assimilation preprocessing tasks. #725

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1455cec
Untested first pass at defining the early workflow
christinaholtNOAA Apr 11, 2023
65808fb
Use cycledefs_type correctly.
christinaholtNOAA Apr 24, 2023
a63f1f0
Set up cycledefs for prod in test config.
christinaholtNOAA Apr 24, 2023
78963a9
Adding entities.
christinaholtNOAA Apr 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions parm/wflow/da_data_preproc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# This group contains all the tasks needed for preprocessing tasks for
# RRFS DA.

default_data_preproc_task: &default_preproc
account: '&ACCOUNT;'
attrs:
cycledefs: #cycle_type#
maxtries: '1'
envars: &default_envars
GLOBAL_VAR_DEFNS_FP: '&GLOBAL_VAR_DEFNS_FP;'
USHdir: '&USHdir;'
PDY: !cycstr "@Y@m@d"
cyc: !cycstr "@H"
subcyc: !cycstr "@M"
LOGDIR: !cycstr "&LOGDIR;"
CYCLE_TYPE: '#cycle_type#'
native: '{{ platform.SCHED_NATIVE_CMD }}'
nodes: '{{ nnodes }}:ppn={{ ppn }}'
nnodes: 1
nodesize: "&NCORES_PER_NODE;"
ppn: 1
partition: '{% if platform.get("PARTITION_DEFAULT") %}&PARTITION_DEFAULT;{% else %}None{% endif %}'
queue: '&QUEUE_DEFAULT;'
walltime: 00:25:00

metatask_process_obs_cycle_type:
var:
cycledefs_type: prodcyc,prodcyc_long
cycle_type: prod

task_process_radarref_#cycle_type#:
<<: *default_preproc
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_RADARREF"'
ppn: 24
dependency:
or:
taskdep:
attrs:
task: get_da_obs_radar
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These names are placeholders for what comes of the retrieve data efforts underway by @mkavulich.

I am leaving the option here to either have this task run, or to have the script know where to look on disk for the sake of efficiency in the real-time runs.

and:
not:
taskvalid:
attrs:
task: get_da_obs_radar
streq:
left: do_real_time
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}'
timedep: '<cyclestr offset="&START_TIME_NSSLMOSAIC;">@Y@m@d@H@M00</cyclestr>'


task_process_lightning_#cycle_type#:
<<: *default_preproc
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_LIGHTNING"'
dependency:
or:
taskdep:
attrs:
task: get_da_obs_lightning
and:
not:
taskvalid:
attrs:
task: get_da_obs_lightning
streq:
left: do_real_time
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}'
timedep: '<cyclestr offset="&START_TIME_LIGHTNING;">@Y@m@d@H@M00</cyclestr>'


task_process_bufrobs_#cycle_type#:
<<: *default_preproc
command: '&LOAD_MODULES_RUN_TASK_FP; "process_obs" "&JOBSdir;/JREGIONAL_PROCESS_BUFROBS"'
join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
dependency:
or:
taskdep:
attrs:
task: get_da_obs_bufr
and:
not:
taskvalid:
attrs:
task: get_da_obs_bufr
streq:
left: do_real_time
right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}'
timedep: '<cyclestr offset="&START_TIME_CONVENTIONAL;">@Y@m@d@H@M00</cyclestr>'

14 changes: 14 additions & 0 deletions tests/WE2E/test_configs/data_assimilation/config.process_obs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
metadata:
description: |-
Tests the data retrieval and data preprocessing tasks that are
precursors to running any data assimialtion components. Datasets
include bufr, lightning, and radar reflectivity.

workflow:
PREDEF_GRID_NAME: RRFS_CONUS_25km
DATE_FIRST_CYCL: '2019061500'
DATE_LAST_CYCL: '2019061500'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll need some feedback from @EdwardSnyder-NOAA on what data has been staged here before this information can be nailed down.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I placed the predefined grid data on Hera, Jet, and Orion for the following grids: RRFS_NA_3km, RRFS_NA_13km, and RRFS_CONUS_3km. The test lightning data is stored on Jet and Orion for the 2022-07-20 00Z case. We still need radar and buffer obs data for that date. Before I add those obs data, I just want to make sure that this is the case date we want to go with. My thinking was that this case could be the sample case we provide to the community for the release.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EdwardSnyder-NOAA I'm sorry for the delay on this one.

I think we should pose this question at a release meeting. More generally "is there an interesting case(s) that we should use for the release?"


rocoto:
tasks:
taskgroups: '{{ ["parm/wflow/da_data_preproc.yaml"]|include }}'