improve ui for neon script #1467

jedwards4b · 2021-08-23T17:07:27Z

Description of changes

Redo options list to remove positional arguments that were difficult to input correctly.
Transient runs now use run_type startup and get finidat from s3 server unless --run-from-postad option is used (or finidat is not available). Use mpi instead of mpi-serial, this mod was recommended for container use. Add a new script neon_finidat_upload which allows authorized users to upload finidat files to the s3 server.

Specific notes

Contributors other than yourself, if any: @ekluzek

CTSM Issues Fixed (include github issue #):
Fixes #1563
Fixes #1550
Fixes #1574

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? Yes for NEON

Testing performed, if any: regular testing and tools testing will be done
cheyenne: -- PASS
izumi: -- OK

tools/site_and_regional/run_neon.py

wwieder · 2021-08-24T14:08:08Z

Jim, this looks good. The only thing that may be helpful is some comments in the code, help functions, or a README that explains what the workflow and how users should use this script.

tools/site_and_regional/run_neon.py

negin513 · 2021-08-24T12:02:16Z

tools/site_and_regional/run_neon.py

+                        [default: %(default)s]
+                        ''',
+                        choices = ["ad", "postad", "transient", "sasu"],
+                        default = "transient")


I like the implementation of specifying the run_type here better than how it was before. Thanks for that.
My only concern is the options that are specific for each run type... If we can generalize the options enough so it works for all run_types, I like this implementation much better.

negin513 · 2021-08-24T14:38:23Z

Thanks @jedwards4b. The changes look great.
Overall, I think we need to modify the instructions on the top of the code to clarify its use for different cases.

This will be part of the README for our future use.

negin513 · 2021-08-24T14:43:22Z

When I am running the code, I receive the following error without any meaningful help:
I think we should set the defaults somehow so the code works for some default cases.
@ekluzek might have a better insight on this but it seems like the CTSM SE team like the codes to work with some defaults.

>> ./run_neon.py

Traceback (most recent call last):
  File "./run_neon.py", line 607, in <module>
    main(__doc__) 
  File "./run_neon.py", line 577, in main
    site_list, output_root, run_type, overwrite, run_length, base_case_root, run_from_postad = get_parser(sys.argv, description, valid_neon_sites)
  File "./run_neon.py", line 213, in get_parser
    return neon_sites, args.output_root, args.run_type, args.overwrite, run_length, base_case_root, args.run_from_postad
UnboundLocalError: local variable 'base_case_root' referenced before assignment

Again, I think this looks good.
I think it will look great if we add what arguments are optional and what arguments are not necessary...or make it work with some defaults...

wwieder · 2021-08-24T14:53:51Z

cime_config/usermods_dirs/NEON/defaults/user_nl_clm

@@ -19,7 +19,7 @@
 !----------------------------------------------------------------------------------

 flanduse_timeseries = ' '   ! This isn't needed for a non transient case, but will be once we start using transient compsets
-fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210720.nc"
+fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210805.nc"
 model_year_align_urbantv = 2018
 stream_year_first_urbantv = 2018
 stream_year_last_urbantv = 2019


Don't we want ndep and popdens to update to the last year of the simulation (or at least the spinup)? this would be 2020 for spinup and 2021 for transient cases. How do we keep updating this moving forward?

@jedwards4b do these get updated somehow below?

@wwieder these don't get autoupdated by any means. Right now the only way to change it is to keep changing the last year in this file (or in each case run). The last year update could also be handedl in run_neon.py though. I don't know that's any better though.

I think this is fine for now, but as we move to making more of the real-time hindcasts they should be updated for new years of data? Updating urban likely ins't too important and for short runs the deposition files shouldn't changethat much?

ekluzek · 2021-08-24T18:06:39Z

@wwieder added some notes about leap years. The model CAN run with leap years when flipped to GREGORIAN calendar mode. Just need to

./xmlchange CALENDAR=GREGORIAN

This could be added as a default setting for NEON sites. I actually thought we already were doing that, but it looks like we aren't yet. Should that be done?

wwieder · 2021-08-24T18:58:30Z

For transient simulations I think we should use ./xmlchange CALENDAR=GREGORIAN, this is critical for aligning results with the visualization script.
For spinup let's keep using noleap, or what ever @jedwards4b is doing to ignore leap days and facilitate this now.

ekluzek · 2021-12-09T21:25:18Z

bld/CLMBuildNamelist.pm

@@ -4254,7 +4254,7 @@ sub check_input_files {
                my $pathname = $nl->get_variable_value($group, $var);
                # Need to strip the quotes
                $pathname =~ s/['"]//g;
-
+                next if ($pathname =~ /UNSET$/);


I also did work in 017a7ed to prevent blank files being listed. I think @jedwards4b is also a good protection although I think it really should trigger an error because something is screwy if you have a namelist item set to UNSET.

The comment that generated this was...

#1198 (comment)

ekluzek · 2021-12-09T21:29:50Z

cime_config/usermods_dirs/NEON/GUAN/shell_commands

@@ -1,3 +1,4 @@
 ./xmlchange NEONSITE=GUAN
 ./xmlchange PTS_LON=293.13112
 ./xmlchange PTS_LAT=17.96882
+./xmlchange RUN_STARTDATE=2018-06-01


@wwieder It will just start on June/1st and use that as it's reference date. This means for purposes of spinup dynamics June/1st is the date when spinup variables are updated. It's sort of like making June/1st the start of your fiscal year. Somethings are still tied to the calendar year (like dynamic land use change), but anything that is measuring time since start of run or doing running averages will use June/1st as the marker. Does that help?

ekluzek · 2021-12-09T21:33:08Z

cime_config/usermods_dirs/NEON/defaults/user_nl_clm

@@ -19,7 +19,7 @@
 !----------------------------------------------------------------------------------

 flanduse_timeseries = ' '   ! This isn't needed for a non transient case, but will be once we start using transient compsets
-fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210720.nc"
+fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210805.nc"
 model_year_align_urbantv = 2018
 stream_year_first_urbantv = 2018
 stream_year_last_urbantv = 2019


@wwieder these don't get autoupdated by any means. Right now the only way to change it is to keep changing the last year in this file (or in each case run). The last year update could also be handedl in run_neon.py though. I don't know that's any better though.

wwieder

These changes look good

wwieder · 2021-12-09T22:57:01Z

@ekluzek I have one outstanding question about how the model handles spinup when data are not available Jan 1 for a particular year, but otherwise I think my questions have been addressed for this PR.

jedwards4b · 2021-12-09T23:20:44Z

@wwieder consider the log file here: /glade/scratch/jedwards/NEON/archive/GUAN.ad/logs/atm.log.290175.chadmin1.ib0.cheyenne.ucar.edu.210901-142537 It starts the spinup at model date 180101 and with ATM data file GUAN_atm_2020-01.nc and spins up using 2019 and 2020 data only. This is controlled by DATM_YR_START which is set via variable self.start_year in the run_neon.py script using the first year for which JAN 1 data is available.

wwieder · 2021-12-10T13:10:13Z

thanks for clarifying @jedwards4b

…_steps

… script and exclude the pytlint file, also have pylint ignore an option that black uses so they are compatable

wwieder · 2021-12-10T23:19:19Z

sorry, one more on these lines

model_year_align_urbantv = 2018
stream_year_first_urbantv = 2018
stream_year_last_urbantv = 2019
...

Should all of these at least go to 2020 for the spinups, which is what we're doing now? They'll get out of date for the transient runs, but I'm less worried about this (for now), as it will be a while untile we have many years of data past 2020

ekluzek · 2021-12-10T23:25:44Z

@wwieder do all sites have data that goes through 2020 now?

The would be problems with cases that don't have 2020 data as the case will point out the missing a files. Bit that could be considered a feature not a big...

wwieder · 2021-12-10T23:43:40Z

Yep we're (mostly) spinning all sites up by cycling over 2018-2020 data and then running a full 'transient' from 2018-2021

…s/neon_updates

…to jedwards/neon_updates

ekluzek · 2021-12-11T00:49:31Z

OK @wwieder I changed the end year to 2020 in commit 411c321 and also brought in @negin513 update of surface datasets from #1539.

I've run the standard testing at this point. The other thing I'm going to do is run the tools testing and make the ChangeLog and then this tag will be done.

…master list

… a config file for it

…n error code up the chain and the code can die rather than going on as if an error didn't happen

…negative longitude this fixes ESCOMP#1574

jedwards4b added 5 commits August 18, 2021 11:08

improve ui for neon script

6b35998

fix run_length issues

73a4164

do not use mpi-serial

eaa0e0e

update surface files

231658f

more run_neon improvements

0010273

jedwards4b requested review from ekluzek, negin513 and wwieder August 23, 2021 17:07

jedwards4b assigned jedwards4b and unassigned jedwards4b Aug 23, 2021

wwieder reviewed Aug 24, 2021

View reviewed changes

changes prompeted by code review

9c67f30

negin513 reviewed Aug 24, 2021

View reviewed changes

wwieder added the enhancement new capability or improved behavior of existing capability label Aug 24, 2021

This was linked to issues Aug 24, 2021

Spinup: Create NEON initial conditions files #1385

Closed

Wrapper: Batch run single point sites, including NEON #1354

Closed

initialize base_case_root

68b0059

wwieder reviewed Aug 24, 2021

View reviewed changes

This was linked to issues Aug 24, 2021

write out friction velocity to history files by default #1382

Closed

History output of CTSM data that NEON will be hosting #1376

Closed

Change run_neon to run from any directory and add some additional options for testing #1459

Closed

add --setup-only and --no-batch options

e5ecb46

billsacks mentioned this pull request Aug 24, 2021

write out friction velocity to history files by default #1382

Closed

wwieder removed a link to an issue Aug 24, 2021

write out friction velocity to history files by default #1382

Closed

ekluzek added this to the ctsm5.1.0 milestone Dec 9, 2021

ekluzek approved these changes Dec 9, 2021

View reviewed changes

wwieder closed this Dec 9, 2021

wwieder reopened this Dec 9, 2021

wwieder approved these changes Dec 9, 2021

View reviewed changes

ekluzek added 2 commits December 10, 2021 11:43

Get balance check unit test on skip steps working with change in skip…

a06a059

…_steps

Change the call of black from a reformat to a check, incldue the main…

abb3b30

… script and exclude the pytlint file, also have pylint ignore an option that black uses so they are compatable

negin513 and others added 4 commits December 10, 2021 17:08

updated neon surface dataset

b407a37

Update last year to 2020 as we have data through the end of 2020 now

411c321

Merge remote-tracking branch 'negin513/neon_modify_surf' into jedward…

bee0da5

…s/neon_updates

Merge branch 'jedwards/neon_updates' of github.com:jedwards4b/CTSM in…

304990b

…to jedwards/neon_updates

ekluzek added 8 commits December 11, 2021 00:38

Remove tests from the test lists that were removed from the master list

abe76e9

Remove smf tests from tests lists as they have been removed from the …

e9953fc

…master list

Add note about running a baseline compare case

e043f53

Move explicit settings for black and list of includes and excludes to…

9afbad8

… a config file for it

Change stop statements into calls to abort() so that they will pass a…

07214b4

…n error code up the chain and the code can die rather than going on as if an error didn't happen

Replace SCRIP grid file and mapping files for 1x1_brazil as they had …

6ef9999

…negative longitude this fixes ESCOMP#1574

Start updating the Change files

a5b873a

Update of Change files

ae1a023

ekluzek merged commit 9d56452 into ESCOMP:master Dec 13, 2021

ekluzek deleted the jedwards/neon_updates branch December 13, 2021 07:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve ui for neon script #1467

improve ui for neon script #1467

jedwards4b commented Aug 23, 2021 •

edited by ekluzek

Loading

wwieder commented Aug 24, 2021

negin513 Aug 24, 2021

negin513 commented Aug 24, 2021

negin513 commented Aug 24, 2021 •

edited

Loading

wwieder Aug 24, 2021

wwieder Aug 26, 2021

ekluzek Dec 9, 2021

wwieder Dec 9, 2021

ekluzek commented Aug 24, 2021

wwieder commented Aug 24, 2021

ekluzek Dec 9, 2021

ekluzek Dec 9, 2021

ekluzek Dec 9, 2021

wwieder left a comment

wwieder commented Dec 9, 2021

jedwards4b commented Dec 9, 2021

wwieder commented Dec 10, 2021

wwieder commented Dec 10, 2021

ekluzek commented Dec 10, 2021

wwieder commented Dec 10, 2021

ekluzek commented Dec 11, 2021

improve ui for neon script #1467

improve ui for neon script #1467

Conversation

jedwards4b commented Aug 23, 2021 • edited by ekluzek Loading

Description of changes

Specific notes

wwieder commented Aug 24, 2021

Choose a reason for hiding this comment

negin513 commented Aug 24, 2021

negin513 commented Aug 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekluzek commented Aug 24, 2021

wwieder commented Aug 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwieder left a comment

Choose a reason for hiding this comment

wwieder commented Dec 9, 2021

jedwards4b commented Dec 9, 2021

wwieder commented Dec 10, 2021

wwieder commented Dec 10, 2021

ekluzek commented Dec 10, 2021

wwieder commented Dec 10, 2021

ekluzek commented Dec 11, 2021

jedwards4b commented Aug 23, 2021 •

edited by ekluzek

Loading

negin513 commented Aug 24, 2021 •

edited

Loading