Fixes missing `$` in variable reference in archive job.

Resolves NOAA-EMC#2022

This PR addresses issue NOAA-EMC#1130. A new task `postanl.sh` has been added such that the
`anl` post-processing is not it's own task within the Rocoto workflow. This change
prevents the GDAS experiment from breaking due to failures caused by the `anl` not
being available for the first cycle.

Resolves NOAA-EMC#1130

)

This PR will move the GSI monitor tasks from the `vrfy` job into their own separate jobs in
the rocoto mesh. This takes care of part of issue NOAA-EMC#235.

Changes in this PR:

1. Create new individual GSI monitor jobs in the rocoto mesh to match how it is done in
operations. The new jobs are: `gdasverfozn`, `gdasverfrad`, `gdasvminmon`, and `gfsvminmon`.
The names correspond to the names in operations and the ecf script names. The jobs are only
added to cycled mode, none of the jobs are run in forecast-only mode.
2. Updates to GSI monitor scripts:
  * Update `ush/jjob_header.sh` arguments in JJOB scripts to call new individual job names and
configs.
  * Introduce `COM_*` variables and use of the `generate_com` utility. This results in the outputs
landing in `COM` subfolders, like occurs in operations, instead of landing in the online archive.
  * Some variable cleanup in JJOB and ush scripts, includes cleaning up usage of `DATA` variable
and its removal at the end of the jobs.
  * Add missing exit statements to `JGFS_ATMOS_VMINMON` and `JGDAS_ATMOS_VERFRAD`.
3. Create new rocoto job scripts for the new individual GSI monitor jobs.
4. Create new configs for the new individual GSI monitor jobs.
5. Add new individual GSI monitor jobs to resource configuration, using the same resources as
in operations.
6. Move and rename GSI monitor job switches from `config.vrfy` to `config.base`. Also add
switches to setup system. New switches to control each job being added to the rocoto mesh
are: `DO_VERFOZN`, `DO_VERFRAD`, `DO_VMINMON`
7. Add GSI monitor job output to archival job and into resulting HPSS tarballs or local tarballs.
This now matches what is done in operations.
8. Remove the GSI monitor tasks from the rocoto `vrfy` job and associated `config.vrfy`.

Example output locations and files (not showing contents of oznmon/horiz or oznmon/time for
brevity, see archival script for full list of output files):
```
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gdas.20211221/00/products/atmos/*mon
gdas.20211221/00/products/atmos/minmon:
total 112
-rw-r--r-- 1 kate.friedman emc 14030 Oct 25 15:00 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 84932 Oct 25 15:00 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 15:00 gnorm_data.txt

gdas.20211221/00/products/atmos/oznmon:
total 8
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 horiz
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 time

gdas.20211221/00/products/atmos/radmon:
total 21036
-rw-r--r-- 1 kate.friedman emc      231 Oct 25 15:12 bad_diag.2021122100
-rw-r--r-- 1 kate.friedman emc     9035 Oct 25 15:12 bad_pen.2021122100
-rw-r--r-- 1 kate.friedman emc     1449 Oct 25 15:12 low_count.2021122100
-rw-r--r-- 1 kate.friedman emc 20523403 Oct 25 15:07 radmon_angle.tar.gz
-rw-r--r-- 1 kate.friedman emc   217272 Oct 25 15:08 radmon_bcoef.tar.gz
-rw-r--r-- 1 kate.friedman emc   502151 Oct 25 15:10 radmon_bcor.tar.gz
-rw-r--r-- 1 kate.friedman emc   264480 Oct 25 15:12 radmon_time.tar.gz
-rw-r--r-- 1 kate.friedman emc      684 Oct 25 15:12 warning.2021122100
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gfs.20211221/00/products/atmos/minmon/
total 88
-rw-r--r-- 1 kate.friedman emc 10530 Oct 25 14:55 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 63882 Oct 25 14:55 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 14:55 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   608 Oct 25 14:55 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 14:55 gnorm_data.txt
```

Lingering work and considerations:

1. Issue NOAA-EMC#1925
2. Consider not hardcoding the list of types in archival and perhaps tie it to something reliable or
configurable (maybe into `config.verfozn`?):
`subtyplist="gome_metop-b omi_aura ompslp_npp ompsnp_n20 ompsnp_npp ompstc8_n20 ompstc8_npp sbuv2_n19"`

Resolves NOAA-EMC#1908

During the update to stage_ic, the copying of the additional res_N files for
0p25 was omitted. These are now properly copied.

Resolves NOAA-EMC#2027

…S2SW job hanging issue on WCOSS2 (NOAA-EMC#2021)

…A-EMC#1944)

Allows cycling in IAU mode with the S2S UFS model, soca and the GSI.

Most of the work related to making IAU work was done by @JessicaMeixner-NOAA . Thanks @JessicaMeixner-NOAA !

Resolves NOAA-EMC#1943 
Refs NOAA-EMC#1776

Cleaned up the logs for CI PRs sent to the GitHub Messages:

- Eliminates redundant accumulative reporting
- Reports on all created or skipped experiment instantiations in a single message
- Gives one line report once  for each experiment completion
- Single final report completion of CI

See some examples in forked [PR 195](TerrenceMcGuinness-NOAA#195)

Resolves NOAA-EMC#2007

Reduce the default number of build jobs for the GDASApp to 8 from 16.

This is needed for Orion as the build crashes during a linking step.  Though not verified, it
appears this may be caused by using too much memory with 16 builds.  The issue disappears
when using 8 build jobs.

Resolves NOAA-EMC#2029

This PR updates the `ufs-weather-model` to the latest hash that included the dust fix (PR NOAA-EMC#1922).
Along with this I removed the nitrates by default in support of the GEFSv13 EP4 and EP5
experiments. Removed unneeded inputs that should help with speed improvements.

…y_hr3

Conflicts:
	Externals.cfg
	sorc/checkout.sh