Merge GFS v16.3 operational gsi scripts into develop branch #1158

lgannoaa · 2022-12-08T15:33:58Z

Changes were made in the GSI scripts for the operational system that need to be brought back to development:

Remove hard-wired rCDUMP in EnKF forecast job
Turn on cao_check and ta2tb
Update cloud coefficient fix file

Fixes #1148

My code follows the style guidelines of this project
I have performed a self-review of my own code

GitHub# 1148

WalterKolczynski-NOAA

Looks fine to me, pending testing and DA group approval.

WalterKolczynski-NOAA · 2022-12-12T06:37:57Z

Why was so much of the PR template removed (the entire testing section and much of the checklist that would be relevant)?

RussTreadon-NOAA

I was not involved in the details of the GFS v16.3 implementation. Emily was the DA POC. The changes in this PR appear to be correct but I can not vouch their correctness.

aerorahul · 2023-01-06T16:43:03Z

@emilyhcliu
Can you please review this so we can proceed?

emilyhcliu · 2023-01-06T16:44:47Z

Rahul, I am reviewing it now. Emily *Emily Huichun Liu* Physical Scientist Modeling and Data Assimilation Branch - DA & QC NOAA/NWS/NCEP Environmental Modeling Center National Centers for Weather and Climate Prediction (NCWCP) 5830 University Research Court College Park, MD 20740 Phone: 301-683-3639 Cell: 410-948-6256

…

On Fri, Jan 6, 2023 at 11:43 AM Rahul Mahajan ***@***.***> wrote: @emilyhcliu <https://github.com/emilyhcliu> Can you please review this so we can proceed? — Reply to this email directly, view it on GitHub <#1158 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AITLO5RTKSU53WY6LUUFKQTWRBDSHANCNFSM6AAAAAASYI5BXY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

emilyhcliu

The changes look good to me.

XianwuXue-NOAA · 2023-01-10T04:24:54Z

jobs/JGDAS_ENKF_FCST

@@ -52,7 +52,6 @@ status=$?
 export CDATE=${CDATE:-${PDY}${cyc}}
 export CDUMP=${CDUMP:-${RUN:-"gdas"}}
 export COMPONENT="atmos"
-export rCDUMP="gdas"


This hard coded line is for gfsefmn jobs in early enkf. This will cause jobs can not find the data in previous cycle enkf/gdas

You are right @XianwuXue-NOAA
My apologies for missing this in the review.
We can open a hotfix asap.

No problem. the gdas[gfs]anal[eobs] jobs also failed, after reverse the changes in exglobal_atmos_analysis.sh, I can run it successfully. I tried to change crtm_ver from 2.3.0 to 2.4.0, however, it does not pass the value. I think I modified the wrong file (ecf/versions/gfs.ver).

The initialize of the rCDUMP is for rCDUMP used in forecast_predet.sh that is default to CDUMP.
A GW meeting has decision made to revert this change.

Update: When I change the cram version from 2.3.0 to 2.4.0 in modulefiles/module_base.wcoss2.lua, then gdas[gfs]anal[eobs] jobs works with the exglobal_atmos_analysis.sh changes in this PR.

WalterKolczynski-NOAA · 2023-01-10T09:50:08Z

I'm also experiencing issues related to this PR when trying to run my test case. I think it is from the fix file change. I'm inclined to revert until we can reassess and redo. Can discuss at tag-up.

WalterKolczynski-NOAA · 2023-01-12T18:13:47Z

Reverting this until the issues related to it can be resolved (crtm version and early cycle EnKF).

…1158)" This reverts commit d07db2b.

…1158)" (#1238) Reverts #1158 due to issues it creates due to crtm version mismatches that can't be immediately corrected.

* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)

Merge GFS v16.3 operational gsi.fd job and script into develop branch.

d4cf4fd

GitHub# 1148

lgannoaa self-assigned this Dec 8, 2022

lgannoaa requested review from aerorahul and emilyhcliu December 8, 2022 15:34

lgannoaa marked this pull request as ready for review December 8, 2022 15:53

WalterKolczynski-NOAA changed the title ~~Merge GFS v16.3 operational gsi.fd job and script into develop branch.~~ Merge GFS v16.3 operational gsi scripts into develop branch Dec 12, 2022

WalterKolczynski-NOAA approved these changes Dec 12, 2022

View reviewed changes

aerorahul requested a review from RussTreadon-NOAA January 5, 2023 19:15

RussTreadon-NOAA reviewed Jan 5, 2023

View reviewed changes

emilyhcliu approved these changes Jan 6, 2023

View reviewed changes

WalterKolczynski-NOAA merged commit d07db2b into NOAA-EMC:develop Jan 9, 2023

XianwuXue-NOAA reviewed Jan 10, 2023

View reviewed changes

WalterKolczynski-NOAA mentioned this pull request Jan 10, 2023

Abstract common j-job tasks #1230

Merged

14 tasks

WalterKolczynski-NOAA added a commit that referenced this pull request Jan 12, 2023

Revert "Merge GFS v16.3 operational GSI changes into develop branch. (#…

ecd7a19

…1158)" This reverts commit d07db2b.

WalterKolczynski-NOAA mentioned this pull request Jan 12, 2023

Revert "Merge GFS v16.3 operational gsi scripts into develop branch" #1238

Merged

WalterKolczynski-NOAA added a commit that referenced this pull request Jan 12, 2023

Revert "Merge GFS v16.3 operational GSI changes into develop branch. (#…

55667e9

…1158)" (#1238) Reverts #1158 due to issues it creates due to crtm version mismatches that can't be immediately corrected.

KateFriedman-NOAA mentioned this pull request Jan 24, 2023

GFS v16.3.5 implementation NOAA-EMC/GSI#511

Closed

CatherineThomas-NOAA mentioned this pull request Apr 20, 2023

Include script changes from v16.3 upgrade again #1494

Closed

KateFriedman-NOAA mentioned this pull request Apr 26, 2023

Redo v16.3 GSI script updates to scripts/exglobal_atmos_analysis.sh #1535

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge GFS v16.3 operational gsi scripts into develop branch #1158

Merge GFS v16.3 operational gsi scripts into develop branch #1158

lgannoaa commented Dec 8, 2022 •

edited by WalterKolczynski-NOAA

Loading

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA commented Dec 12, 2022

RussTreadon-NOAA left a comment

aerorahul commented Jan 6, 2023

emilyhcliu commented Jan 6, 2023 via email

emilyhcliu left a comment

XianwuXue-NOAA Jan 10, 2023

aerorahul Jan 10, 2023

XianwuXue-NOAA Jan 10, 2023

lgannoaa Jan 10, 2023

XianwuXue-NOAA Jan 10, 2023

WalterKolczynski-NOAA commented Jan 10, 2023

WalterKolczynski-NOAA commented Jan 12, 2023

Merge GFS v16.3 operational gsi scripts into develop branch #1158

Merge GFS v16.3 operational gsi scripts into develop branch #1158

Conversation

lgannoaa commented Dec 8, 2022 • edited by WalterKolczynski-NOAA Loading

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Dec 12, 2022

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

aerorahul commented Jan 6, 2023

emilyhcliu commented Jan 6, 2023 via email

emilyhcliu left a comment

Choose a reason for hiding this comment

XianwuXue-NOAA Jan 10, 2023

Choose a reason for hiding this comment

aerorahul Jan 10, 2023

Choose a reason for hiding this comment

XianwuXue-NOAA Jan 10, 2023

Choose a reason for hiding this comment

lgannoaa Jan 10, 2023

Choose a reason for hiding this comment

XianwuXue-NOAA Jan 10, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Jan 10, 2023

WalterKolczynski-NOAA commented Jan 12, 2023

lgannoaa commented Dec 8, 2022 •

edited by WalterKolczynski-NOAA

Loading