Wrapper script for batch running NEON sites, and NEON fixes

[negin513]

Description of changes

This is a wrapper script for running CTSM simulation for one or more neon sites using CIME case objects.

Based on @ekluzek and @jedwards4b suggestions, I rewrote the wrapper script in #1426 to use CIME case objects.

This script accomplishes the following:

• Downloads and parses neon listing file for the available neon data :
  https://neon-ncar.s3.data.neonscience.org/listing.csv

• Create a base generic case and clone other cases from this generic base case.

• Enable the user to run in AD, Post-AD, and transient mode.

Currently, there is no case dependency or case status check in the code. Which means that the user cannot run Post-AD +AD + transient all at the same time. If we add case dependency, then the code that checks the status of a case (e.g. spinup) every 2 seconds and once it is finished successfully submit the next phase (e.g. transient.)

Externals are also updated to fix some NEON issues.

New tools subdirectory site_and_regional and current scripts for handling single-point moved underneath. Also README files and tools testing changed for this move. Also tools test added for the new wrapper script talked about above. Fix an issue in subset_data.py where it couldn't be run from outside of a git directory because it checks for the git hash.

Changes so that CTSM will write restarts as signaled at end of simulation by the CMEPS mediator.

Specific notes

Contributors other than yourself, if any: @wwieder , @danicalombardozzi , @jedwards4b

CTSM Issues Fixed (include github issue #):
Fixes #1354
Fixes #1454
Fixes #1456
Fixes #1407
Fixes #1446
Fixes #1429
Some work on #1441 to create new tools subdirectory and move some things around as well as updating README files

Are answers expected to change (and if so in what way)? NO

Any User Interface Changes (namelist or namelist defaults changes)? No

This script has the following interface:

|------------------------------------------------------------------|
|---------------------  Instructions  -----------------------------|
|------------------------------------------------------------------|
This is a wrapper script for running CTSM simulation for one or more
neon sites. 

This script is only for neon site and we will develop a more general
code later.

This script first creates and builds a generic base case. 
Next, it will clone the base_case for different neon sites and run
types to reduce the need to build ctsm everytime. 

This script will do the following:
    1) Create a generic base case for cloning. 
    2) Make the case for the specific neon site(s).
    3) Make changes to the case, for:
        a. AD spinup
	b. post-AD spinup
        c. transient
    	#---------------
    	d. SASU or Matrix spinup --> not working
    4) Build and submit the case.
 
-------------------------------------------------------------------
Instructions for running on Cheyenne/Casper:
 
load the following into your local environment
    module load python
    ncar_pylib
 
To remove NPL from your environment on Cheyenne/Casper:
    deactivate
 
-------------------------------------------------------------------
To see the available options:
    ./run_neon.py --help
-------------------------------------------------------------------

positional arguments:
  {ad,postad,transient,sasu}
                        Four different ways to run this script.
    ad                  AD spin-up options
    postad              Post-AD spin-up options
    transient           Transient spin-up options
    sasu                Sasu spin-up options --not in CTSM yet

optional arguments:
  -h, --help            show this help message and exit
  --neon_site {ABBY,BARR,BART,BLAN,BONA,CLBJ,CPER,DCFS,DEJU,DELA,DSNY,GRSM,GUAN,HARV,HEAL,JERC,JORN,KONA,KONZ,LAJA,LENO,MLBS,MOAB,NIWO,NOGP,OAES,ONAQ,ORNL,OSBS,PUUM,RMNP,SCBI,SERC,SJER,SOAP,SRER,STEI,STER,TALL,TEAK,TOOL,TREE,UKFS,UNDE,WOOD,WREF,YELL} [{ABBY,BARR,BART,BLAN,BONA,CLBJ,CPER,DCFS,DEJU,DELA,DSNY,GRSM,GUAN,HARV,HEAL,JERC,JORN,KONA,KONZ,LAJA,LENO,MLBS,MOAB,NIWO,NOGP,OAES,ONAQ,ORNL,OSBS,PUUM,RMNP,SCBI,SERC,SJER,SOAP,SRER,STEI,STER,TALL,TEAK,TOOL,TREE,UKFS,UNDE,WOOD,WREF,YELL} ...]
                        4-letter neon site code.
  --surf_dir SURF_DIR   Directory of single point surface dataset. [default:
                        /glade/scratch/negins/single_point/]
  --out_dir OUT_DIR     Directory to write updated single point surface dataset.
                        [default: /glade/scratch/negins/neon_sims/]
  -a, --all             Flag for running all neon sites, instead of selected few sites.
                        [default: False]
  -d, --debug           Debug mode will print more information.
-----------------
  ad 
                        run ad case
  --ad_length AD_LENGTH
                        How many years to run AD spin-up [default: 200]

-----------------
  postad 
                        run postad case
  --postad_length POSTAD_LENGTH
                        How many years to run in post-AD mode [default: 100]
  --finidat FINIDAT_POSTAD
                        finidat file location from spinup step to start from. [default:
                        None]
-----------------
  transient 
                        run transient case
  --start_year START_YEAR
                        Start year for running CTSM simulation. [default: 2018]
  --end_year END_YEAR   End year for running CTSM simulation. [default: 2020]
  --finidat FINIDAT_TRANSIENT
                        finidat file location from spinup step to start from. [default:
                        None]

[wwieder]

@negin513 should we close #1426 now?

[billsacks]

I like the use of cime case objects here. Thanks for your work on this @negin513 !

Once this is integrated with the other ctsm python modules, you can make use of https://github.com/ESCOMP/CTSM/blob/master/python/ctsm/path_utils.py and/or https://github.com/ESCOMP/CTSM/blob/master/python/ctsm/add_cime_to_path.py to add the cime lib to your path in what I think is a robust way. (I'm open to suggestions if you can see a better way to do this.)

[wwieder]

After talking with @jedwards4b and @danicalombardozzi we decided that it's not worth the effort at this time to have the script automatically run AD, postAD, and transient simulations for single point cases. While this functionality is desirable in the long-term for CLM developers, it's not really critical for the NCAR-NEON application that we're focusing on now.

[ekluzek]

Standard testing has gone fine, with the exception of this test: ERS_Ly3_P72x2_Vnuopc.f10_f10_mg37.IHistClm50BgcCropG.cheyenne_intel.clm-cropMonthOutput
which fails at run time...

[jedwards4b]

@ekluzek I think you are using the wrong externals here
cime should be cime6.0.5
cmeps should be cmeps0.13.22
cdeps is still to be tagged but will be cdeps0.12.19

[jedwards4b]

@ekluzek Erik please update cmeps to cmeps0.13.23.

[ekluzek]

OK I updated externals as advised but still had trouble with the above IG test (but now in CISM) until I backed off to cime6.0.4 where testing all worked as expected.