Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix shell check and merge develop #2

Merged
merged 12 commits into from
Nov 7, 2023
Merged

Fix shell check and merge develop #2

merged 12 commits into from
Nov 7, 2023

Conversation

JessicaMeixner-NOAA
Copy link

Description

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

HenryRWinterbottom and others added 12 commits November 2, 2023 13:18
@HenryRWinterbottom
Corrects missing local variable reference (NOAA-EMC#2023)
247cdf8 
Fixes missing `$` in variable reference in archive job.

Resolves NOAA-EMC#2022
@HenryRWinterbottom
Move analysis post to new stand-alone task (NOAA-EMC#1979)
31b05a9 
This PR addresses issue NOAA-EMC#1130. A new task `postanl.sh` has been added such that the
`anl` post-processing is not it's own task within the Rocoto workflow. This change
prevents the GDAS experiment from breaking due to failures caused by the `anl` not
being available for the first cycle.

Resolves NOAA-EMC#1130
@KateFriedman-NOAA
Create separate GSI monitor jobs and remove from vrfy job (NOAA-EMC#1983
93bc918 
)

This PR will move the GSI monitor tasks from the `vrfy` job into their own separate jobs in
the rocoto mesh. This takes care of part of issue NOAA-EMC#235.

Changes in this PR:

1. Create new individual GSI monitor jobs in the rocoto mesh to match how it is done in
operations. The new jobs are: `gdasverfozn`, `gdasverfrad`, `gdasvminmon`, and `gfsvminmon`.
The names correspond to the names in operations and the ecf script names. The jobs are only
added to cycled mode, none of the jobs are run in forecast-only mode.
2. Updates to GSI monitor scripts:
  * Update `ush/jjob_header.sh` arguments in JJOB scripts to call new individual job names and
configs.
  * Introduce `COM_*` variables and use of the `generate_com` utility. This results in the outputs
landing in `COM` subfolders, like occurs in operations, instead of landing in the online archive.
  * Some variable cleanup in JJOB and ush scripts, includes cleaning up usage of `DATA` variable
and its removal at the end of the jobs.
  * Add missing exit statements to `JGFS_ATMOS_VMINMON` and `JGDAS_ATMOS_VERFRAD`.
3. Create new rocoto job scripts for the new individual GSI monitor jobs.
4. Create new configs for the new individual GSI monitor jobs.
5. Add new individual GSI monitor jobs to resource configuration, using the same resources as
in operations.
6. Move and rename GSI monitor job switches from `config.vrfy` to `config.base`. Also add
switches to setup system. New switches to control each job being added to the rocoto mesh
are: `DO_VERFOZN`, `DO_VERFRAD`, `DO_VMINMON`
7. Add GSI monitor job output to archival job and into resulting HPSS tarballs or local tarballs.
This now matches what is done in operations.
8. Remove the GSI monitor tasks from the rocoto `vrfy` job and associated `config.vrfy`.

Example output locations and files (not showing contents of oznmon/horiz or oznmon/time for
brevity, see archival script for full list of output files):
```
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gdas.20211221/00/products/atmos/*mon
gdas.20211221/00/products/atmos/minmon:
total 112
-rw-r--r-- 1 kate.friedman emc 14030 Oct 25 15:00 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 84932 Oct 25 15:00 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 15:00 gnorm_data.txt

gdas.20211221/00/products/atmos/oznmon:
total 8
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 horiz
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 time

gdas.20211221/00/products/atmos/radmon:
total 21036
-rw-r--r-- 1 kate.friedman emc      231 Oct 25 15:12 bad_diag.2021122100
-rw-r--r-- 1 kate.friedman emc     9035 Oct 25 15:12 bad_pen.2021122100
-rw-r--r-- 1 kate.friedman emc     1449 Oct 25 15:12 low_count.2021122100
-rw-r--r-- 1 kate.friedman emc 20523403 Oct 25 15:07 radmon_angle.tar.gz
-rw-r--r-- 1 kate.friedman emc   217272 Oct 25 15:08 radmon_bcoef.tar.gz
-rw-r--r-- 1 kate.friedman emc   502151 Oct 25 15:10 radmon_bcor.tar.gz
-rw-r--r-- 1 kate.friedman emc   264480 Oct 25 15:12 radmon_time.tar.gz
-rw-r--r-- 1 kate.friedman emc      684 Oct 25 15:12 warning.2021122100
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gfs.20211221/00/products/atmos/minmon/
total 88
-rw-r--r-- 1 kate.friedman emc 10530 Oct 25 14:55 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 63882 Oct 25 14:55 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 14:55 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   608 Oct 25 14:55 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 14:55 gnorm_data.txt
```

Lingering work and considerations:

1. Issue NOAA-EMC#1925
2. Consider not hardcoding the list of types in archival and perhaps tie it to something reliable or
configurable (maybe into `config.verfozn`?):
`subtyplist="gome_metop-b omi_aura ompslp_npp ompsnp_n20 ompsnp_npp ompstc8_n20 ompstc8_npp sbuv2_n19"`

Resolves NOAA-EMC#1908
@AndrewEichmann-NOAA
changed optional script name vars for ocean analysis (NOAA-EMC#2025)
bd4c56d
@WalterKolczynski-NOAA
Fix staging of MOM data (NOAA-EMC#2028)
cbdc097 
During the update to stage_ic, the copying of the additional res_N files for
0p25 was omitted. These are now properly copied.

Resolves NOAA-EMC#2027
@jiandewang
Add two ucx modules in load_ufswm_modules.sh to solve C768 and C1152 …
2563806 
…S2SW job hanging issue on WCOSS2 (NOAA-EMC#2021)
@guillaumevernieres
Implement IAU Cycling Type with Marine Data Assimilation Enabled (NOA…
8d55126 
…A-EMC#1944)

Allows cycling in IAU mode with the S2S UFS model, soca and the GSI.

Most of the work related to making IAU work was done by @JessicaMeixner-NOAA . Thanks @JessicaMeixner-NOAA !

Resolves NOAA-EMC#1943 
Refs NOAA-EMC#1776
@TerrenceMcGuinness-NOAA
Streamline CI reporting to PRs (NOAA-EMC#2026)
75269e4 
Cleaned up the logs for CI PRs sent to the GitHub Messages:

- Eliminates redundant accumulative reporting
- Reports on all created or skipped experiment instantiations in a single message
- Gives one line report once  for each experiment completion
- Single final report completion of CI

See some examples in forked [PR 195](TerrenceMcGuinness-NOAA#195)

Resolves NOAA-EMC#2007
@DavidHuber-NOAA
Reduce gdas builds to 8 NOAA-EMC#2029 (NOAA-EMC#2036)
34a73cf 
Reduce the default number of build jobs for the GDASApp to 8 from 16.

This is needed for Orion as the build crashes during a linking step.  Though not verified, it
appears this may be caused by using too much memory with 16 builds.  The issue disappears
when using 8 build jobs.

Resolves NOAA-EMC#2029
@bbakernoaa
Update UFS for dust fix and remove nitrates by default (NOAA-EMC#1989)
5183c43 
This PR updates the `ufs-weather-model` to the latest hash that included the dust fix (PR NOAA-EMC#1922).
Along with this I removed the nitrates by default in support of the GEFSv13 EP4 and EP5
experiments. Removed unneeded inputs that should help with speed improvements.
@JessicaMeixner-NOAA
fix shellcheck
83058ad
@JessicaMeixner-NOAA
Merge remote-tracking branch 'EMC/develop' into mergedev_develop_mdto…
59718ce 
…y_hr3

Conflicts:
	Externals.cfg
	sorc/checkout.sh
@JessicaMeixner-NOAA JessicaMeixner-NOAA mentioned this pull request Nov 7, 2023
Merged
7 tasks
@mdtoyNOAA mdtoyNOAA merged commit 21c7874 into mdtoyNOAA:develop_mdtoy_hr3 Nov 7, 2023
3 checks passed
@JessicaMeixner-NOAA JessicaMeixner-NOAA deleted the mergedev_develop_mdtoy_hr3 branch March 3, 2024 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.