Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Add a workflow for the data assimilation preprocessing tasks. #725

Conversation

christinaholtNOAA
Copy link
Collaborator

DESCRIPTION OF CHANGES:

This PR is still a work in process as it requires stitching together the results of 4 other PRs:

I am opening this one early to facilitate discussion about the direction of the workflow development for the RRFS project.

Here, I am adding untested YAML files that loosely follows what the end product for this PR will look like (hopefully). One YAML defines the default settings needed for these three data preprocessing tasks, then a second test YAML defines what it should look like to pull the data and run the tasks. There are still several details that need to be worked out concerning data retrieval.

Modifications are also planned in this PR to adopt the naming conventions set in Issue #633, but won't be made until the outstanding PRs are merged to develop, and then into this branch.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

TESTS CONDUCTED:

  • hera.intel
  • orion.intel
  • cheyenne.intel
  • cheyenne.gnu
  • gaea.intel
  • jet.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • Work In Progress
  • bug
  • enhancement
  • documentation
  • release
  • high priority
  • run_ci
  • run_we2e_fundamental_tests
  • run_we2e_comprehensive_tests
  • Needs Cheyenne test
  • Needs Jet test
  • Needs Hera test
  • Needs Orion test
  • help wanted

CONTRIBUTORS (optional):

or:
taskdep:
attrs:
task: get_da_obs_radar
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These names are placeholders for what comes of the retrieve data efforts underway by @mkavulich.

I am leaving the option here to either have this task run, or to have the script know where to look on disk for the sake of efficiency in the real-time runs.

workflow:
PREDEF_GRID_NAME: RRFS_CONUS_25km
DATE_FIRST_CYCL: '2019061500'
DATE_LAST_CYCL: '2019061500'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll need some feedback from @EdwardSnyder-NOAA on what data has been staged here before this information can be nailed down.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I placed the predefined grid data on Hera, Jet, and Orion for the following grids: RRFS_NA_3km, RRFS_NA_13km, and RRFS_CONUS_3km. The test lightning data is stored on Jet and Orion for the 2022-07-20 00Z case. We still need radar and buffer obs data for that date. Before I add those obs data, I just want to make sure that this is the case date we want to go with. My thinking was that this case could be the sample case we provide to the community for the release.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EdwardSnyder-NOAA I'm sorry for the delay on this one.

I think we should pose this question at a release meeting. More generally "is there an interesting case(s) that we should use for the release?"

yichengt90 pushed a commit to yichengt90/ufs-srweather-app that referenced this pull request Apr 17, 2023
…community#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).
mkavulich added a commit to mkavulich/ufs-srweather-app that referenced this pull request Apr 24, 2023
@christinaholtNOAA
Copy link
Collaborator Author

The contents of this PR was incorporated into @mkavulich's branch for testing observation data retrieval.

@christinaholtNOAA christinaholtNOAA deleted the data_preproc_workflow branch December 8, 2023 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants