[develop] Split calling of METplus's Pb2nc tool into separate task (from PointStat and EnsembleStat tasks)

[gsketefian]

DESCRIPTION OF CHANGES:

Split calls to METplus's Pb2nc tool out of the tasks that run

1. deterministic verification (vx) using the PointStat tool,
2. ensemble vx using the EnsembleStat, and
3. verification on the ensemble mean and ensemble probabilities using the PointStat tool

into separate tasks. This new task is run for each cycle to convert NDAS prep-bufr files to NetCDF format. Also, include additional cleanup of vx tasks.

Details of introduction of Pb2nc task:

• Introduce a new j-job (JREGIONAL_RUN_MET_PB2NC_OBS) and an ex-script (exregional_run_met_pb2nc_obs.sh) to run the new Pb2nc task.
• Introduce a METplus configuration file (Pb2nc_obs.conf) for running the Pb2nc task.
• Remove the capability to run Pb2nc as well as all METplus variables used by Pb2nc from those METplus configuration files for SFC and UPA that are used by the deterministic PointStat (PointStat_[SFC|UPA].conf), EnsembleStat (EnsembleStat_[SFC|UPA].conf), and ensemble mean and probabilistic PointStat (PointStat_ens[mean|prob]_[SFC|UPA].conf) tasks.
• Include thePb2nc task in the ROCOTO template XML and add it to the dependencies of the deterministic PointStat and (ensemble) EnsembleStat tasks for SFC and UPA.
• Add a Pb2nc task section to config_defaults.yaml. Also, in the verification section of this file, add template parameters for NDAS obs files needed in running the Pb2nc task.
• For both deterministic and ensemble PointStat tasks, remove the variable FHR (from the ROCOTO XML, ex-scripts, and METplus configuration files). Instead, in the ex-scripts, define the alternate variable FHR_LIST that is generated by a call to set_vx_fhr_list.sh and use it in the METplus configuration files. This is necessary to take into account missing obs (i.e. FHR_LIST will not include forecast hours for which the obs file is missing). In a future PR, this will also be done for other vx tasks.
• For clarity, in the ROCOTO XML vx tasks and vx ex-scripts, change variable names as follows:
  • Change ACCUM to ACCUM_HH to emphasize that it represents a 2-digit hour.
  • Change SLASH_ENSMEM_SUBDIR to SLASH_ENSMEM_SUBDIR_OR_NULL and use it for both deterministic and ensemble forecasts (where for deterministic forecasts, this gets set to an empty string).
• In the ROCOTO XML, add the variable ACCUM_HH (and set it to a null value) because it is needed in the call to set_vx_fhr_list.sh (although it is not used). This makes it easier to later collapse the various PointStat tasks in the ROCOTO XML into a loop.

Other changes/cleanup not directly related to the new Pb2nc task:

• Move the LOG_DIR variable from some of the METplus tool-specific configuration files to the shared (among all METplus tools) common.conf config file since LOG_DIR is defined in the same way in all the tool-specific files. In a future PR, LOG_DIR will be removed from any remaining tool-specific configuration files that still contain it.
• Change vx task output directory names from
  pb2nc_obs, pcp_combine_[obs|fcst], grid_stat, point_stat, ensemble_stat
  etc, to
  Pb2nc_obs, PcpCombine_[obs|fcst], GridStat, PointStat, EnsembleStat.
• For consistency with other vx tasks, in the EnsembleStat tasks don't include ensgrid_ in name of the "final" METplus configuration file (metplus_final...).
• Change names of output directories generated by the ensemble mean and probabilistic GridStat and PointStat tasks so that instead of containing the (mis-named) string ensemble_stat_, they contain the string GridStat_ or PointStat_.
• For clarity, change the strings _mean and _prob in METplus output file and directory names to _ensmean and _ensprob, respectively.
• Remove unneeded PCP_COMBINE variable settings from (both deterministic and ensemble) GridStat METplus configuration files for APCP01h, REFC, and RETOP.
• For vx tasks that run on the SFC set of fields, change the strings conus_surface and conus_sfc in vx output directory and file names to SFC.
• For vx tasks that run on the UPA set of fields, change the string upper_air in vx output directory and file names to UPA.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

The following set of fundamental tests were conducted on Hera with the Intel compiler. All were successful.

MET_verification
MET_verification_only_vx
community_ensemble_2mems
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot
MET_ensemble_verification
community_ensemble_2mems_stoch
pregen_grid_orog_sfc_climo

DOCUMENTATION:

Documentation is needed but will be added once the whole set of vx PRs has been merged (see Issue #630).

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[danielabdi-noaa]

Can we make sure MET_ensemble_verification works with this PR in NCO mode if possible?
I was getting the following error on the run_MET_PcpCombine_fcst_APCP03h_mem002 task

+ exregional_run_met_pcpcombine.sh[230]: print_err_msg_exit 'The list of forecast hours for which to run METplus is empty:
  FHR_LIST = []'

[danielabdi-noaa]

To make nco mode work

Suggested change

	<join><cyclestr>&LOGDIR;/{{tn}}_@Y@m@d@H.log</cyclestr></join>
	<join>&LOGDIR;/{{tn}}_<cyclestr>@Y@m@d@H</cyclestr>&LOGEXT;</join>

[JeffBeck-NOAA]

@danielabdi-noaa, thanks for pointing this out!

[gsketefian]

@danielabdi-noaa Thanks for pointing this out. I changed it, but there are other issues in NCO mode that prevent the tasks working in that mode (mostly paths need to be set correctly). I will address them in my next PR.

[MichaelLueken]

@gsketefian These changes look good to me! I have left a few comments regarding commented out lines in some of the METplus configuration files and ex- scripts.

[gsketefian]

@MichaelLueken I removed commented out lines except in the two ex-script that I'd like to keep for reference until my next PR. Will remove them in that one.

I retested with vx WE2E tests and everything still works (but not NCO mode, for which I'll put i a fix in my next PR). So if you and @danielabdi-noaa are ok with this, it's ready for automated tests. Thanks.

[MichaelLueken]

@gsketefian Thanks for addressing my concerns! Approving now.

[MichaelLueken]

@gsketefian All Jenkins tests successfully passed. On Orion, the tests sat in queue for several hours (over fourteen hours), leading to a time-out for the Build and Test check. Moving forward with your PR now.

[gsketefian]

@MichaelLueken Thanks Mike. I will try to get my next PR opened today.