-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
First pass at subsetter.py #93
Merged
Merged
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
6cd9e6b
Start on subsetter script.
davidfillmore f98130a
Form subsetted file names.
davidfillmore 170349f
Added required_vars YAML entry.
davidfillmore 32c2e9a
Start on AirNow v WRF-Chem subset example.
davidfillmore 49a0b4c
Start on AirNow v WRF-Chem subset example.
davidfillmore f4e642d
Point to correct YAML file.
davidfillmore 1d7c67e
Use /Users/davidfillmore env variable.
davidfillmore a9c7526
Added path substitution with env vars to subsetter.
davidfillmore f88a5a3
Changed fnames to file_str.
davidfillmore 3797a51
Use os.path.expandvars.
davidfillmore fdbadbf
Use HOME path in YAML file.
davidfillmore 71fde54
Merge branch 'develop_envvar' into develop_subsetter
davidfillmore c0d5502
Added license header.
dwfncar f43440a
Prepend _ to subsetter script name.
dwfncar ba51a05
Merge branch 'develop' into develop_subsetter
davidfillmore 44f8031
Moved subsetter.py script to new scripts subdir.
dwfncar cb78b36
Added CARMA config to subset branch.
dwfncar bca22d8
Added optional outdir argument.
dwfncar 6225d9d
Make required_vars optional in YAML model blocks.
dwfncar e3a2466
Save.
dwfncar accddca
Merge branch 'develop' into develop_subsetter
dwfncar 227fd1c
Update scripts/subsetter.py
dwfncar aba8cad
Update scripts/subsetter.py
dwfncar 075115a
Update scripts/subsetter.py
dwfncar a7c34d5
Update examples/subset/run.py
dwfncar 37d36b0
Update scripts/subsetter.py
dwfncar 63aa76d
Update examples/subset/run.py
dwfncar 57807bf
Update examples/subset/run.py
dwfncar d26a0dc
Update scripts/subsetter.py
dwfncar 476d013
Added comments for needed PR modifications.
davidfillmore 635a80f
Merge branch 'develop' into develop_subsetter
dwfncar c2de9e7
Added compression level argument.
dwfncar 3bb142c
Fixed os.path.join of subdirs.
dwfncar 90fdb45
Merge branch 'develop' into develop_subsetter
dwfncar ea85898
Update examples/subset/airnow_wrfchem.yaml
dwfncar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
analysis: | ||
start_time: '2019-09-05-06:00:00' #UTC | ||
end_time: '2019-09-06-06:00:00' #UTC | ||
output_dir: $HOME/Data/melodies_monet_output | ||
debug: True | ||
|
||
|
||
model: | ||
# Regional Atmospheric Chemistry Model/Earth System Research Lab | ||
WRF-Chem: | ||
# files: $HOME/Data/wrfchem_example/racm_esrl/wrfout_d01_2019* | ||
files: $HOME/Data/wrfchem_example/racm_esrl/subset_wrfout_d01_2019* | ||
required_vars: | ||
- Times | ||
- HGT | ||
- PSFC | ||
- P | ||
- PB | ||
- PH | ||
- PHB | ||
- XTIME | ||
- XLAT | ||
- XLONG | ||
- T | ||
- QVAPOR | ||
mod_type: 'wrfchem' | ||
mod_kwargs: | ||
mech: 'racm_esrl_vcp' | ||
radius_of_influence: 12000 # meters | ||
mapping: # model species name : obs species name | ||
AirNow: | ||
PM2_5_DRY: "PM2.5" | ||
o3: "OZONE" | ||
projection: ~ | ||
plot_kwargs: | ||
color: 'magenta' | ||
marker: 's' | ||
linestyle: '-' | ||
|
||
|
||
obs: | ||
AirNow: | ||
use_airnow: True | ||
filename: $HOME/Data/surface_obs/AIRNOW_20190901_20190930.nc | ||
obs_type: pt_sfc | ||
variables: | ||
PM2.5: | ||
unit_scale: 1 | ||
unit_scale_method: '*' | ||
nan_value: -1.0 | ||
ylabel_plot: 'PM2.5 (ug/m3)' | ||
ty_scale: 2.0 | ||
vmin_plot: 0.0 | ||
vmax_plot: 22.0 | ||
vdiff_plot: 15.0 | ||
nlevels_plot: 23 | ||
OZONE: | ||
unit_scale: 1 | ||
unit_scale_method: '*' | ||
nan_value: -1.0 | ||
ylabel_plot: 'Ozone (ppbv)' | ||
vmin_plot: 15.0 | ||
vmax_plot: 55.0 | ||
vdiff_plot: 20.0 | ||
nlevels_plot: 21 | ||
|
||
|
||
plots: | ||
|
||
plot_timeseries_group: | ||
type: 'timeseries' | ||
fig_kwargs: | ||
figsize: [12, 6] | ||
default_plot_kwargs: | ||
linewidth: 2.0 | ||
markersize: 10 | ||
text_kwargs: | ||
fontsize: 24 | ||
domain_type: ['all','state_name','epa_region'] | ||
domain_name: ['CONUS','CA','R9'] | ||
data: ['AirNow_WRF-Chem'] | ||
data_proc: | ||
rem_obs_nan: True | ||
ts_select_time: 'time_local' | ||
ts_avg_window: 'H' | ||
set_axis: True | ||
|
||
plot_overlay_group: | ||
type: 'spatial_overlay' | ||
fig_kwargs: | ||
states: True | ||
figsize: [10, 5] | ||
text_kwargs: | ||
fontsize: 16 | ||
domain_type: ['all','state_name','epa_region'] | ||
domain_name: ['CONUS','CA','R9'] | ||
data: ['AirNow_WRF-Chem'] | ||
data_proc: | ||
rem_obs_nan: True | ||
set_axis: True | ||
|
||
plot_bias_group: | ||
type: 'spatial_bias' | ||
fig_kwargs: | ||
states: True | ||
figsize: [10, 5] | ||
text_kwargs: | ||
fontsize: 16 | ||
domain_type: ['all','state_name','epa_region'] | ||
domain_name: ['CONUS','CA','R9'] | ||
data: ['AirNow_WRF-Chem'] | ||
data_proc: | ||
rem_obs_nan: True | ||
set_axis: True |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
analysis: | ||
start_time: '2019-06' | ||
end_time: '2019-08' | ||
freq: 'MS' | ||
output_dir: $HOME/Plots/CARMA | ||
|
||
obs: | ||
MOD08_M3: | ||
data_format: gridded_eos | ||
datadir: $HOME/Data/MOD08_M3 | ||
filestr: MOD08_M3.AYYYYDDD.061.*.hdf | ||
# filestr: MOD08_M3.AYYYYDDD.061.*.nc | ||
variables: | ||
AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean: | ||
fillvalue: -9999 | ||
scale: 0.001 | ||
units: none | ||
plot_params: | ||
name: 'Terra MODIS AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.7 | ||
nlevel: 15 | ||
|
||
model: | ||
MAM4: | ||
# datadir: /glade/campaign/acom/acom-climate/tilmes/CARMA/... | ||
data_format: cesm | ||
datadir: $HOME/Data/CARMA/FCnudged_f09.mam.mar27.2000_2021.001/atm/hist | ||
filestr: FCnudged_f09.mam.mar27.2000_2021.001.cam.h0.YYYY-MM.nc | ||
variables: | ||
AODdnMODE1: | ||
plot_params: | ||
name: 'MAM4 Accumulation Mode AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.7 | ||
nlevel: 15 | ||
AODdnMODE2: | ||
plot_params: | ||
name: 'MAM4 Aitken Mode AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.1 | ||
nlevel: 11 | ||
AODdnMODE3: | ||
plot_params: | ||
name: 'MAM4 Coarse Mode AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.7 | ||
nlevel: 15 | ||
AODVIS: | ||
plot_params: | ||
name: 'MAM4 AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.7 | ||
nlevel: 15 | ||
combine: | ||
AODVIS: | ||
- AODdnMODE1 | ||
- AODdnMODE2 | ||
- AODdnMODE3 | ||
mapping: | ||
MOD08_M3: | ||
AODVIS: AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean | ||
|
||
CARMA: | ||
# datadir: /glade/scratch/tilmes/archive/ | ||
datadir: $HOME/Data/CARMA/FCnudged_f09.carma_trop_stratv05.2000_2020.001/atm/hist | ||
data_format: cesm | ||
filestr: FCnudged_f09.carma_trop_stratv05.2000_2020.001.cam.h0.YYYY-MM.nc | ||
variables: | ||
AODVIS: | ||
plot_params: | ||
name: 'CARMA AOD 550 nm YYYY M_ABBR' | ||
range_min: 0.0 | ||
range_max: 0.7 | ||
nlevel: 15 | ||
mapping: | ||
MOD08_M3: | ||
AODVIS: AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean | ||
|
||
|
||
# plots: | ||
# spatial: | ||
# monthly_climo: | ||
|
||
# time_series: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
from melodies_monet import driver | ||
|
||
an = driver.analysis() | ||
an.control = 'airnow_wrfchem.yaml' | ||
an.read_control() | ||
|
||
an.open_models() | ||
|
||
an.open_obs() | ||
|
||
an.pair_data() | ||
|
||
an.plotting() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,73 @@ | ||||||
# Copyright (C) 2022 National Center for Atmospheric Research and National Oceanic and Atmospheric Administration | ||||||
# SPDX-License-Identifier: Apache-2.0 | ||||||
# | ||||||
import os | ||||||
import sys | ||||||
import subprocess | ||||||
import argparse | ||||||
import logging | ||||||
import yaml | ||||||
from glob import glob | ||||||
|
||||||
parser = argparse.ArgumentParser() | ||||||
parser.add_argument('--control', type=str, | ||||||
default='control.yaml', | ||||||
help='yaml control file') | ||||||
parser.add_argument('--logfile', type=str, | ||||||
default=sys.stdout, | ||||||
help='log file (default: stdout)') | ||||||
parser.add_argument('--outdir', type=str, | ||||||
default=None, | ||||||
help='output directory') | ||||||
parser.add_argument('--compress_level', type=int, | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Personally, I prefer this style, but up to you. |
||||||
default=1, | ||||||
help='NetCDF compression level') | ||||||
parser.add_argument('--debug', action='store_true', | ||||||
help='set logging level to debug') | ||||||
args = parser.parse_args() | ||||||
|
||||||
# Setup logging | ||||||
logging_level = logging.DEBUG if args.debug else logging.INFO | ||||||
logging.basicConfig(stream=args.logfile, level=logging_level) | ||||||
|
||||||
# Read YAML control | ||||||
with open(args.control, 'r') as f: | ||||||
control = yaml.safe_load(f) | ||||||
|
||||||
logging.debug(control) | ||||||
|
||||||
# set compression level | ||||||
level_str = str(args.compress_level) | ||||||
|
||||||
for model in control['model']: | ||||||
logging.info('processing:' + model) | ||||||
|
||||||
var_str = '-v ' | ||||||
|
||||||
if 'required_vars' in control['model'][model]: | ||||||
for var in control['model'][model]['required_vars']: | ||||||
var_str += var + ',' | ||||||
|
||||||
for var in control['model'][model]['variables']: | ||||||
var_str += var + ',' | ||||||
|
||||||
var_str = var_str.strip(',') | ||||||
Comment on lines
+51
to
+54
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like could use |
||||||
logging.info(var_str) | ||||||
|
||||||
file_str = os.path.expandvars(control['model'][model]['files']) | ||||||
|
||||||
files = sorted(glob(file_str)) | ||||||
print(files) | ||||||
|
||||||
for file_in in files: | ||||||
file_subdirs = list(os.path.split(file_in)) | ||||||
file_subdirs[-1] = 'subset_' + file_subdirs[-1] | ||||||
if args.outdir is not None: | ||||||
file_out = os.path.join(args.outdir, file_subdirs[-1]) | ||||||
else: | ||||||
file_out = os.path.join(*tuple(file_subdirs)) | ||||||
command = f'ncks -O -L {level_str} {var_str} {file_in} {file_out}' | ||||||
logging.info(command) | ||||||
# os.system(command) | ||||||
subprocess.run(command.split()) | ||||||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.