Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process JEDI radiance files as a single tarball #2862

Closed
RussTreadon-NOAA opened this issue Aug 27, 2024 · 3 comments · Fixed by #2875
Closed

Process JEDI radiance files as a single tarball #2862

RussTreadon-NOAA opened this issue Aug 27, 2024 · 3 comments · Fixed by #2875
Assignees
Labels
feature New feature or request

Comments

@RussTreadon-NOAA
Copy link
Contributor

What new functionality do you need?

At present JEDI radiance bias correction uses three files per satellite_sensor. This result in many individual files to move, copy, link, archive. While JEDI applications need to work with individual files, file operations are more easily applied to a single file.

This can be achived by tar'ing all JEDI radiance bias correction files into a single file. This single file is what gets moved, copied, linked, and archived. Individual radiance bias correction files can be obtained by untar'ing the radiance tarball in the run directory of specific jobs.

What are the requirements for the new functionality?

Update JEDI DA scripts which currently process individual radiance bias correction files to process a single radiance tarball.

Acceptance Criteria

Once work for this issue is complete, the COM directory will contain a single radiance bias correction file per cycle. Analysis results when using this tarball should be identical to results obtained by using individual radiance bias correction files.

Suggest a solution (optional)

No response

@RussTreadon-NOAA RussTreadon-NOAA added feature New feature or request triage Issues that are triage labels Aug 27, 2024
@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Aug 27, 2024
@RussTreadon-NOAA
Copy link
Contributor Author

Work for this issue will be done in RussTreadon-NOAA:feature/radbcor

@RussTreadon-NOAA
Copy link
Contributor Author

Hera g-w C96C48_ufs_hybatmDA CI

Install feature/radbcor at 20fb88c on Hera. Set up g-w CI for C96C48_ufs_hybatmDA.

Note, for these test specify ICSDIR_ROOT=/scratch1/NCEPDEV/global/Russ.Treadon/data/ICSDIR. This is copy of the glopara ICSDIR_ROOT with the satbias and tlapse radiance bias correction files for C96C48_ufs_hybatmDA replaced with a single gdas.t18z.radbcor tarball.

All enkfgdas, gdas, and gfs DA jobs successfully run to completion. Only 20240224 00Z gfs forecast and post jobs remain running.

The enkfgdas, gdas, and gfs init jobs extracted satellite_sensor radiance bias correction files from the radbcor tarball. The final job created a radbcor tarball with updated coefficient and copied the file to the appropriate COM directory. The gdas archive job included the radbcor tarball in gdas_restarta.tar.

This test will be repeated using the official ICSDIR_ROOT once issue #2869 is closed.

@RussTreadon-NOAA
Copy link
Contributor Author

GDASApp PR #1267 updates the drivers for the atmospheric local ensemble and variational ctests to process the radiance bias correction tarball instead of individual satellite_sensor bias correction files. The sorc/gdas.cd hash in g-w feature/radbcor will be updated a new GDASApp hash once PR #1267 is approved and merged into GDASApp develop.

RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Aug 29, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the triage Issues that are triage label Sep 3, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 7, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 9, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 10, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 10, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 11, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 12, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 17, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 18, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 19, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 20, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Sep 27, 2024
WalterKolczynski-NOAA pushed a commit that referenced this issue Sep 30, 2024
)

This PR updates g-w components which handle JEDI radiance bias
correction files. Currently, JEDI radiance bias correction files are
processed at the _satellite_sensor_. This PR replaces these multiple
files with a single radiance bias correction tarball.

Resolves #2862
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants