Feature/gfsv16intocoupled

[JianKuang-Intelsat]

Script update.

Additional code and logics blocks from gfsv16 are included in the modularized scripts.

Next step is to add them into the namelists

[JessicaMeixner-NOAA]

@JianKuang-NOAA the run on hera failed the same way my run on orion failed. See the log file

/work/noaa/marine/jmeixner/p6pr194update/test01/COMROOT/test01/logs/2013040100/gfs.forecast.highres.log

Important parts:

0 + source /work/noaa/marine/jmeixner/p6pr194update/global-workflow/scripts/../ush/parsing_model_configure_FV3.sh
/work/noaa/marine/jmeixner/p6pr194update/global-workflow/scripts/../ush/parsing_model_configure_FV3.sh: line 76: syntax error near unexpected token `}'
/work/noaa/marine/jmeixner/p6pr194update/global-workflow/scripts/../ush/parsing_model_configure_FV3.sh: line 76: `}'

forrtl: severe (29): file not found, unit 101, file /work/noaa/marine/jmeixner/p6pr194update/test01/DATA/test01/2013040100/gfs/fcst.378149/model_configure

[JianKuang-Intelsat]

@JessicaMeixner-NOAA This is likely a typo. It is so strange that my test didn't capture it. Let me check.

[JessicaMeixner-NOAA]

@JianKuang-NOAA sorry I can't do this through the normal review as there are so many files with changes it's hard to go through and mark them on the file. But I just checked the diffs in the input.nml creation from the namelist options that are in ush/parsing_namelists_FV3.sh and we need to update the following to match the develop branch:

These are the variables in the develop branch options that I found that did not match the defaults in this PR:
adjust_dry_mass = ${adjust_dry_mass:-".true."} 
lheatstrg    = ${lheatstrg-".false."}
lgfdlmprad   = ${lgfdlmprad:-".false."}
lgfdlmprad   = ${lgfdlmprad:-".true."}
effr_in      = ${effr_in:-".false."}
do_ugwp      = ${do_ugwp:-".true."} 
do_tofd      = ${do_tofd:-".true."}
reiflag = ${reiflag:-"2"}

In the develop branch cplwav needs to default to false:   
cplwav       = ${cplwav:-".false."}
not just be:
cplwav       = ${cplwav}

The if $CCPP_SUITE blocks starting on line 201 could be updated to just be one if block following the "if [ $RUN_CCPP = "YES" ]; then" if block in the develop branch starting on line 1092

lines 234-5 should be removed:
iovr_lw = ${iovr_lw:-"3"}
iovr_sw = ${iovr_sw:-"3"}

Line 486: 
&nam_sfcperts
should instead be   
$nam_sfcperts_nml

[JessicaMeixner-NOAA]

@JessicaMeixner-NOAA This is likely a typo. It is so strange that my test didn't capture it. Let me check.

Are there local changes that did not get pushed?

[JessicaMeixner-NOAA]

@JianKuang-NOAA Also doing a difference between the develop branch and this PR I saw that
Files develop/sorc/build_ncep_post.sh and pr194/sorc/build_ncep_post.sh differ
which I don't think should necessarily be different.

There were a couple of other files such as :
Files develop/parm/config/config.gldas and pr194/parm/config/config.gldas differ
Files develop/env/HERA.env and pr194/env/HERA.env differ

which I believe is just there from recent commits to the develop branch.

[JianKuang-Intelsat]

@JessicaMeixner-NOAA I fixed the model_configure script issue which is a simple syntax error. I will fix those name list variables that you mentioned, together with it.

[JianKuang-Intelsat]

@JessicaMeixner-NOAA I think the RUN_CCPP block in the develop branch is not so clear because there are multiple CCPP suites and each of them may come with some additional name list variables. In other words, CCPP should be a selectable set instead of a "true/false" variable. Indeed the RUN_CCPP block in the develop only takes care of the "FV3_GSD_v0" suite. Please correct me if I am wrong.

[JianKuang-Intelsat]

@JianKuang-NOAA sorry I can't do this through the normal review as there are so many files with changes it's hard to go through and mark them on the file. But I just checked the diffs in the input.nml creation from the namelist options that are in ush/parsing_namelists_FV3.sh and we need to update the following to match the develop branch:

These are the variables in the develop branch options that I found that did not match the defaults in this PR:
adjust_dry_mass = ${adjust_dry_mass:-".true."} 
lheatstrg    = ${lheatstrg-".false."}
lgfdlmprad   = ${lgfdlmprad:-".false."}
lgfdlmprad   = ${lgfdlmprad:-".true."}
effr_in      = ${effr_in:-".false."}
do_ugwp      = ${do_ugwp:-".true."} 
do_tofd      = ${do_tofd:-".true."}
reiflag = ${reiflag:-"2"}

In the develop branch cplwav needs to default to false:   
cplwav       = ${cplwav:-".false."}
not just be:
cplwav       = ${cplwav}

The if $CCPP_SUITE blocks starting on line 201 could be updated to just be one if block following the "if [ $RUN_CCPP = "YES" ]; then" if block in the develop branch starting on line 1092

lines 234-5 should be removed:
iovr_lw = ${iovr_lw:-"3"}
iovr_sw = ${iovr_sw:-"3"}

Line 486: 
&nam_sfcperts
should instead be   
$nam_sfcperts_nml

Line 486 "nam_sfcperts_nml", is identical with the develop branch. I don't know what should I do?

[JessicaMeixner-NOAA]

@JianKuang-NOAA It's my understanding that we need these for all CCPP suites, not just "FV3_GSD_v0" In the develop branch I think the suite that was recently tested was FV3_GFS_v16beta and we're going to be using the coupled version of that for prototype 6.

[JianKuang-Intelsat]

@JianKuang-NOAA It's my understanding that we need these for all CCPP suites, not just "FV3_GSD_v0" In the develop branch I think the suite that was recently tested was FV3_GFS_v16beta and we're going to be using the coupled version of that for prototype 6.

Yes. That's the reason why I think we shouldn't define CCPP as a YES/NO option. Instead we need to enable users to select suite by it's name.

[JessicaMeixner-NOAA]

@JianKuang-NOAA those options are needed for any CCPP suite at this time. If you want to keep it for a specific suite file, please add an if block for "FV3_GFS_v16beta_coupled" with all of the options in the develop branch in that if block.

You are correct on the nam_sfcperts_nml, it is already how it is in the develop branch. I am not sure what I saw before.

[JessicaMeixner-NOAA]

I pulled the recent changes (up to 14a85cd) and tried again and the forecast still fails this time with:
forrtl: severe (19): invalid reference to variable in NAMELIST input, unit -5, file Internal Formatted NML Read, line -1, position 16

Let me know when you are ready for me to run a test again.

[JianKuang-Intelsat]

@JessicaMeixner-NOAA It turns out that's because I included iovr_* variables in the "general" name list(for every CCPP suite). I fixed this issue. Please take another try on your end. I apologize for this mistake.

[JessicaMeixner-NOAA]

@JianKuang-NOAA the iovr and other namelist options look good. Thanks for adding the CCPP suite options for FV3_GFS_v16beta_coupled as well!

Is the forecast model working again? Is this ready for me to test again on hera?

[JianKuang-Intelsat]

@JessicaMeixner-NOAA Yes it is working on my side again.

[JessicaMeixner-NOAA]

@JianKuang-NOAA my test on orion also succeeded. I started with a fresh clone and build so it's taking a little longer on hera.

@LydiaStefanova-NOAA and @jiandewang could you help check output Lydia note that atm files are now in the "atmos" subfolder
The test on orion output is here: /work/noaa/marine/jmeixner/p6pr194update/test02/COMROOT/test02/gfs.20130401/00 I'll post the hera test as soon as it's available.

[JessicaMeixner-NOAA]

@JianKuang-NOAA if we wanted to get bit-reproducible answers, we could by overwriting the default values.. however since we're in the process of upgrading, perhaps as long as things are running this is not an issue. We do want to make sure that everything is working as expected though.

[JessicaMeixner-NOAA]

@jiandewang @LydiaStefanova-NOAA the hera test output is here:
/scratch2/NCEPDEV/climate/Jessica.Meixner/p6pr194update/test01/COMROOT/test01/gfs.20130401/00

The hera test ran to "succeed" although the archive job is still running. The archive job is not completely correct as @jiandewang pointed out earlier but we can address that in a later PR.

[JessicaMeixner-NOAA]

Copying @LydiaStefanova-NOAA's comment's to the PR from the issue:
In atmos/ the grib2 files look ok to me.

In wave/ the gfswave.t00z.global.0p50.fXXX.grib2 files all fields I checked are constant (i.e. identical value at any lat/lon)

Two other things I noticed, that may or may not be relevant here, mentioning them in case they are:
(a) The variable called "la_mod" in the waveocn_*nc files has possibly strange values (e.g the max is 5.37e+34 in wavocn_2013_04_01_03.nc, 8.9301E+08 in wavocn_2013_04_02_21.nc). I haven't looked at this type of files before, so I don't know if this is normal.
(b) the archlist contents (now under atmos/archlist) are different from what we had for P5. I can describe the differences in more detail if this is relevant now.

And from @jiandewang:
checking ocean outputs show reasonable value (both orion and hera), but confirmed with Lydia's point about LA_MOD

And from me:
We know that the arch list is not perfect and will take care of that later. The bad wave grib files is another known issue that this PR is not supposed to address. the la_mod does have very large values at times. That was there before.

Thanks to @jiandewang and @LydiaStefanova-NOAA for helping check things.

[JessicaMeixner-NOAA]

Thanks @JianKuang-NOAA