-
Notifications
You must be signed in to change notification settings - Fork 312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve ui for neon script #1467
Changes from 11 commits
6b35998
73a4164
eaa0e0e
231658f
0010273
9c67f30
68b0059
e5ecb46
39d46e7
4d0b046
8e145cd
234bf61
d28115c
098501d
8d90743
cdc3a5e
11cb158
955afec
d8d9af8
1770da5
f8bfd17
235f524
4a0373f
3af0b34
b5661af
a82688c
93dd98d
7527f6d
b53c809
25ab14f
aa95f0d
99f7812
7b352c6
157a2ef
5afa96c
bd92fd4
99a85fe
b807e8e
91edfeb
c9a99ec
a0349e3
8b514dc
5444c1d
fb2a152
0d69139
7178ea3
405ccbb
a106275
2b5bc4c
bb2181c
06c99c6
b917e02
9963e46
4c1b355
9f767fa
fda1569
9c4c0d6
3d16070
5f00876
a06a059
abb3b30
b407a37
411c321
bee0da5
304990b
abe76e9
e9953fc
e043f53
9afbad8
07214b4
6ef9999
a5b873a
ae1a023
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,7 +19,7 @@ | |
!---------------------------------------------------------------------------------- | ||
|
||
flanduse_timeseries = ' ' ! This isn't needed for a non transient case, but will be once we start using transient compsets | ||
fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210720.nc" | ||
fsurdat = "$DIN_LOC_ROOT/lnd/clm2/surfdata_map/NEON/surfdata_hist_78pfts_CMIP6_simyr2000_${NEONSITE}_c210805.nc" | ||
model_year_align_urbantv = 2018 | ||
stream_year_first_urbantv = 2018 | ||
stream_year_last_urbantv = 2019 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't we want ndep and popdens to update to the last year of the simulation (or at least the spinup)? this would be 2020 for spinup and 2021 for transient cases. How do we keep updating this moving forward? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jedwards4b do these get updated somehow below? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @wwieder these don't get autoupdated by any means. Right now the only way to change it is to keep changing the last year in this file (or in each case run). The last year update could also be handedl in run_neon.py though. I don't know that's any better though. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is fine for now, but as we move to making more of the real-time hindcasts they should be updated for new years of data? Updating urban likely ins't too important and for short runs the deposition files shouldn't changethat much? |
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,173 @@ | ||
#! /usr/bin/env python3 | ||
""" | ||
Script to rename and upload NEON site finidat files for use in transient startup cases | ||
|
||
""" | ||
|
||
import os, sys | ||
# Get the ctsm util tools and then the cime tools. | ||
_CTSM_PYTHON = os.path.abspath(os.path.join(os.path.dirname(__file__), "..","..",'python')) | ||
sys.path.insert(1, _CTSM_PYTHON) | ||
|
||
import boto3 | ||
from botocore.exceptions import ClientError | ||
import glob | ||
import datetime | ||
from ctsm import add_cime_to_path | ||
from ctsm.path_utils import path_to_ctsm_root | ||
from standard_script_setup import * | ||
from CIME.case import Case | ||
from CIME.utils import expect, safe_copy | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
def get_parser(args, description, valid_neon_sites): | ||
""" | ||
Get Parser object for this script | ||
""" | ||
parser = argparse.ArgumentParser(description=description, | ||
formatter_class=argparse.RawDescriptionHelpFormatter) | ||
|
||
CIME.utils.setup_standard_logging_options(parser) | ||
|
||
parser.print_usage = parser.print_help | ||
|
||
parser.add_argument('--neon-sites', | ||
help='4-letter neon site code.', | ||
action="store", | ||
required=False, | ||
choices=valid_neon_sites + ['all'], | ||
dest="neon_sites", | ||
default=["OSBS"], | ||
nargs='+') | ||
|
||
parser.add_argument('--output-root', | ||
help=''' | ||
Root Directory of case results | ||
[default: %(default)s] | ||
''', | ||
action="store", | ||
dest="output_root", | ||
type =str, | ||
required=False, | ||
default=os.getcwd()) | ||
|
||
parser.add_argument('--file-date', | ||
help=''' | ||
Date of ctsm restart file(s) to upload | ||
''', | ||
action="store", | ||
dest="file_date", | ||
required = False, | ||
type = datetime.date.fromisoformat, | ||
default = datetime.datetime.strptime("0268-01-01",'%Y-%m-%d')) | ||
|
||
|
||
parser.add_argument('--upload-finidat', | ||
help=''' | ||
Upload the final restart files from the end of the postad run for each site. | ||
''', | ||
action="store_true", | ||
dest="upload_finidat", | ||
required = False, | ||
default = False) | ||
|
||
parser.add_argument('--upload-history', | ||
help=''' | ||
Upload the transient run h1 history files for each site. | ||
''', | ||
action="store_true", | ||
dest="upload_history", | ||
required = False, | ||
default = False) | ||
|
||
|
||
|
||
args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) | ||
|
||
if 'all' in args.neon_sites: | ||
neon_sites = valid_neon_sites | ||
else: | ||
neon_sites = args.neon_sites | ||
for site in neon_sites: | ||
if site not in valid_neon_sites: | ||
raise ValueError("Invalid site name {}".format(site)) | ||
|
||
expect(args.upload_finidat or args.upload_history,"Must specify at least one of --upload-finidat or --upload-history") | ||
|
||
return neon_sites, args.output_root, args.file_date, args.upload_finidat, args.upload_history | ||
|
||
def upload_file(file_name, bucket, object_name=None): | ||
"""Upload a file to an S3 bucket | ||
|
||
:param file_name: File to upload | ||
:param bucket: Bucket to upload to | ||
:param object_name: S3 object name. If not specified then file_name is used | ||
:return: True if file was uploaded, else False | ||
""" | ||
|
||
# If S3 object_name was not specified, use file_name | ||
if object_name is None: | ||
object_name = os.path.basename(file_name) | ||
|
||
# Upload the file | ||
s3_client = boto3.client('s3') | ||
try: | ||
logger.info("Uploading file {} to {}".format(file_name, object_name)) | ||
response = s3_client.upload_file(file_name, bucket, object_name) | ||
except ClientError as e: | ||
logger.error(e) | ||
return False | ||
return True | ||
|
||
def main(description): | ||
""" | ||
For each site in the site_list find the site.postad run directory and grab the latest clm restart file | ||
from there, | ||
""" | ||
|
||
if not os.path.isfile(os.path.join(os.getenv("HOME"),".aws","credentials")): | ||
raise FileNotFoundError("User account must have valid aws credentials to run this script.") | ||
|
||
cesmroot = path_to_ctsm_root() | ||
# Get the list of supported neon sites from usermods | ||
valid_neon_sites = glob.glob(os.path.join(cesmroot,"cime_config","usermods_dirs","NEON","[!d]*")) | ||
valid_neon_sites = [v.split('/')[-1] for v in valid_neon_sites] | ||
filedatestamp = datetime.datetime.now().date() | ||
site_list, output_root, file_date, upload_finidat, upload_history = get_parser(sys.argv, description, valid_neon_sites) | ||
for site in site_list: | ||
rundir = None | ||
if upload_finidat: | ||
logger.info("Upload finidat for {}".format(site)) | ||
case_path = os.path.join(output_root, site+".postad") | ||
if os.path.isdir(case_path): | ||
with Case(case_path) as case: | ||
rundir = case.get_value("RUNDIR") | ||
basefile = site+".postad.clm2.r.{}-00000.nc".format(file_date.strftime("%4Y-%m-%d")) | ||
finidat_file = os.path.join(rundir,basefile) | ||
if not os.path.isfile(finidat_file): | ||
logger.warning("Could not find file {}".format(finidat_file)) | ||
continue | ||
newfile = basefile.replace(".postad.",".{}.".format(filedatestamp)) | ||
|
||
upload_file(finidat_file, 'neon-ncar-transfer', os.path.join("NEON","lnd","ctsm","initdata",newfile)) | ||
if upload_history: | ||
logger.info("Upload history for {}".format(site)) | ||
case_path = os.path.join(output_root, site+".transient") | ||
if not os.path.isdir(case_path): | ||
logger.warning("No case found in {}".format(case_path)) | ||
continue | ||
with Case(case_path) as case: | ||
archive_dir = os.path.join(case.get_value("DOUT_S_ROOT"),"lnd","hist") | ||
for histfile in glob.iglob(archive_dir + "/*.h1.*"): | ||
newfile = os.path.basename(histfile) | ||
upload_file(histfile, 'neon-ncar-transfer', os.path.join("NEON","archive",site,"lnd","hist",newfile)) | ||
|
||
|
||
|
||
|
||
|
||
|
||
if __name__ == "__main__": | ||
main(__doc__) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ekluzek The change here is to prevent variables marked as UNSET being listed in the ctsm.input_data_list - this should avoid the problem in #1198 wrt fatmlndfrc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is a good start. I'd also like it to not show up in the namelist, as having a file set to "$DIN_LOC_DIR/UNSET" is confusing and looks wrong. I'll likely work on this in addition to what you are doing here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also did work in 017a7ed to prevent blank files being listed. I think @jedwards4b is also a good protection although I think it really should trigger an error because something is screwy if you have a namelist item set to UNSET.
The comment that generated this was...
#1198 (comment)