Wrapper: Batch run single point sites, including NEON

[wwieder]

Modify or create a shell script that can be used to run all of the NEON sites (including spinup and 'transient' cases)?

• Test this at all sites for 5 years (mostly to look for forcing data issues).

• Options for what type of simulations to do including:

  • ADspinup (e.g 200 years),
  • postADspinup (e.g. 50 years), &
  • 'transient' (2018-2020, with high frequency output), starting from 2018 restart files
  • Matrix (SASU) ins't critical now, this could come in later
  • This can draw from Keith’s work for PLUMBER2 /glade/work/oleson/release-clm5.0.34/tools/PTCLM/run_clmtowers_plumber2.csh
  • It can also build on the .xml changes for spinup here /glade/work/wwieder/ctsm/ctsm1.0.dev110/cime/scripts/clm50bgcNEON_ABBY/README_ww.case

• This would be facilitated if NEON data were easier to query to know the length of simulations we can run.

• Versatility of this script should let users:

  • Identify particular sites & years
  • Run all sites for all years

• This greater complexity isn't critical now,

  • Flexibility would let users select options:
  • NEON vs. 'other' sites (e.g. Ameriflux towers where input data are already created and on disk, somewhere)
  • SP vs. BGC simulations (to determine the length of spinup and model configuration).
    • Initial configuration would be NEON - BGC options

[jedwards4b]

I am working on a script that will run multiple sites and allow the user to specify options on the command line. My thought is to include command line options to specify:

• start date
• run length
• site or sites to run (with default all NEON sites with available data)
• ADspinup (boolean)
• BGC (boolean)

My thinking is that at least initially this script would only work for NEON sites and we can revisit this decision once we have something functional.

[wwieder]

@jedwards4b make sure you coordinate with @negin513 to avoid duplicating effort on this one.

[negin513]

Hi @jedwards4b,
I've already written a python wrapper script that accomplishes what you are looking for with the exact same command-line arguments.
The only difference is that in my script I have start date and end date instead of start date and duration.

I will make the PR for this soon.

[jedwards4b]

@negin513 Sounds good, thanks.

[jedwards4b]

@negin513 I would appreciate if you share your work on this so that we can collaborate. I have some ideas and requirements for this tool.

[jedwards4b]

PR #1444 does not yet handle running transient cases from a spun up initial condition. The plan is to add code to the run_neon.py script that will upload the finidat file generated from spinup runs to the
NEON S3 server, these uploaded files will then be moved to the directory
NEON/initdata/v1/NEONSITE/ so that they will be available for download. There will be two initial files available for each site - one for date 2018-01-01 (the beginning of the NEON data period) and the
other for the latest available date. By default the run_neon.py script will run the transient for the most recent period only. A new option will be introduced to run transient from 2018-01-01 with the finidat file (instead of looking for output from the postad as it now does.)

[wwieder]

Discussion today centered on if we update and push new restart files from latest data to NEON for EFI activities?

• Longer term, these real forecasts would be done within the NCAR computing system, not the cloud.
• For this initial effort, let's keep it simple letting the user community tell us what they need for forecasting, DA, and other needs.
• If this is needed, we'd have to build in capabilities to push and pull restart files from NEON-NCAR S3 server which would require some code development and automation.