forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix shell check and merge develop #2
Merged
mdtoyNOAA
merged 12 commits into
mdtoyNOAA:develop_mdtoy_hr3
from
JessicaMeixner-NOAA:mergedev_develop_mdtoy_hr3
Nov 7, 2023
Merged
Fix shell check and merge develop #2
mdtoyNOAA
merged 12 commits into
mdtoyNOAA:develop_mdtoy_hr3
from
JessicaMeixner-NOAA:mergedev_develop_mdtoy_hr3
Nov 7, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes missing `$` in variable reference in archive job. Resolves NOAA-EMC#2022
This PR addresses issue NOAA-EMC#1130. A new task `postanl.sh` has been added such that the `anl` post-processing is not it's own task within the Rocoto workflow. This change prevents the GDAS experiment from breaking due to failures caused by the `anl` not being available for the first cycle. Resolves NOAA-EMC#1130
) This PR will move the GSI monitor tasks from the `vrfy` job into their own separate jobs in the rocoto mesh. This takes care of part of issue NOAA-EMC#235. Changes in this PR: 1. Create new individual GSI monitor jobs in the rocoto mesh to match how it is done in operations. The new jobs are: `gdasverfozn`, `gdasverfrad`, `gdasvminmon`, and `gfsvminmon`. The names correspond to the names in operations and the ecf script names. The jobs are only added to cycled mode, none of the jobs are run in forecast-only mode. 2. Updates to GSI monitor scripts: * Update `ush/jjob_header.sh` arguments in JJOB scripts to call new individual job names and configs. * Introduce `COM_*` variables and use of the `generate_com` utility. This results in the outputs landing in `COM` subfolders, like occurs in operations, instead of landing in the online archive. * Some variable cleanup in JJOB and ush scripts, includes cleaning up usage of `DATA` variable and its removal at the end of the jobs. * Add missing exit statements to `JGFS_ATMOS_VMINMON` and `JGDAS_ATMOS_VERFRAD`. 3. Create new rocoto job scripts for the new individual GSI monitor jobs. 4. Create new configs for the new individual GSI monitor jobs. 5. Add new individual GSI monitor jobs to resource configuration, using the same resources as in operations. 6. Move and rename GSI monitor job switches from `config.vrfy` to `config.base`. Also add switches to setup system. New switches to control each job being added to the rocoto mesh are: `DO_VERFOZN`, `DO_VERFRAD`, `DO_VMINMON` 7. Add GSI monitor job output to archival job and into resulting HPSS tarballs or local tarballs. This now matches what is done in operations. 8. Remove the GSI monitor tasks from the rocoto `vrfy` job and associated `config.vrfy`. Example output locations and files (not showing contents of oznmon/horiz or oznmon/time for brevity, see archival script for full list of output files): ``` kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gdas.20211221/00/products/atmos/*mon gdas.20211221/00/products/atmos/minmon: total 112 -rw-r--r-- 1 kate.friedman emc 14030 Oct 25 15:00 2021122100.costs.txt -rw-r--r-- 1 kate.friedman emc 84932 Oct 25 15:00 2021122100.cost_terms.txt -rw-r--r-- 1 kate.friedman emc 808 Oct 25 15:00 2021122100.gnorms.ieee_d -rw-r--r-- 1 kate.friedman emc 808 Oct 25 15:00 2021122100.reduction.ieee_d -rw-r--r-- 1 kate.friedman emc 80 Oct 25 15:00 gnorm_data.txt gdas.20211221/00/products/atmos/oznmon: total 8 drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 horiz drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 time gdas.20211221/00/products/atmos/radmon: total 21036 -rw-r--r-- 1 kate.friedman emc 231 Oct 25 15:12 bad_diag.2021122100 -rw-r--r-- 1 kate.friedman emc 9035 Oct 25 15:12 bad_pen.2021122100 -rw-r--r-- 1 kate.friedman emc 1449 Oct 25 15:12 low_count.2021122100 -rw-r--r-- 1 kate.friedman emc 20523403 Oct 25 15:07 radmon_angle.tar.gz -rw-r--r-- 1 kate.friedman emc 217272 Oct 25 15:08 radmon_bcoef.tar.gz -rw-r--r-- 1 kate.friedman emc 502151 Oct 25 15:10 radmon_bcor.tar.gz -rw-r--r-- 1 kate.friedman emc 264480 Oct 25 15:12 radmon_time.tar.gz -rw-r--r-- 1 kate.friedman emc 684 Oct 25 15:12 warning.2021122100 kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gfs.20211221/00/products/atmos/minmon/ total 88 -rw-r--r-- 1 kate.friedman emc 10530 Oct 25 14:55 2021122100.costs.txt -rw-r--r-- 1 kate.friedman emc 63882 Oct 25 14:55 2021122100.cost_terms.txt -rw-r--r-- 1 kate.friedman emc 808 Oct 25 14:55 2021122100.gnorms.ieee_d -rw-r--r-- 1 kate.friedman emc 608 Oct 25 14:55 2021122100.reduction.ieee_d -rw-r--r-- 1 kate.friedman emc 80 Oct 25 14:55 gnorm_data.txt ``` Lingering work and considerations: 1. Issue NOAA-EMC#1925 2. Consider not hardcoding the list of types in archival and perhaps tie it to something reliable or configurable (maybe into `config.verfozn`?): `subtyplist="gome_metop-b omi_aura ompslp_npp ompsnp_n20 ompsnp_npp ompstc8_n20 ompstc8_npp sbuv2_n19"` Resolves NOAA-EMC#1908
During the update to stage_ic, the copying of the additional res_N files for 0p25 was omitted. These are now properly copied. Resolves NOAA-EMC#2027
…S2SW job hanging issue on WCOSS2 (NOAA-EMC#2021)
…A-EMC#1944) Allows cycling in IAU mode with the S2S UFS model, soca and the GSI. Most of the work related to making IAU work was done by @JessicaMeixner-NOAA . Thanks @JessicaMeixner-NOAA ! Resolves NOAA-EMC#1943 Refs NOAA-EMC#1776
Cleaned up the logs for CI PRs sent to the GitHub Messages: - Eliminates redundant accumulative reporting - Reports on all created or skipped experiment instantiations in a single message - Gives one line report once for each experiment completion - Single final report completion of CI See some examples in forked [PR 195](TerrenceMcGuinness-NOAA#195) Resolves NOAA-EMC#2007
Reduce the default number of build jobs for the GDASApp to 8 from 16. This is needed for Orion as the build crashes during a linking step. Though not verified, it appears this may be caused by using too much memory with 16 builds. The issue disappears when using 8 build jobs. Resolves NOAA-EMC#2029
This PR updates the `ufs-weather-model` to the latest hash that included the dust fix (PR NOAA-EMC#1922). Along with this I removed the nitrates by default in support of the GEFSv13 EP4 and EP5 experiments. Removed unneeded inputs that should help with speed improvements.
…y_hr3 Conflicts: Externals.cfg sorc/checkout.sh
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Checklist