Merge pull request #290 from NOAA-CSL/develop

Merge Develop to Main

.github/workflows/ci.yml

@@ -17,22 +17,19 @@ jobs:
     strategy:
       matrix:
         monet: [cf, dev]
-      fail-fast: false  # just until cf versions are working
+      fail-fast: false  # always both
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v15
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
-          extra-specs: |
+          cache-environment: true
+          create-args: >-
             pytest
 
-      - name: Install MM
-        run: python -m pip install -e . --no-deps -v
-
       - if: ${{ matrix.monet == 'dev' }}
         name: Install development versions of monet and monetio
         run: |
@@ -49,25 +46,45 @@ jobs:
           python -c "import monet;   print('monet.__version__',   getattr(monet, '__version__', '?'))"
 
       - name: pytest
-        run: python -m pytest melodies_monet
+        run: python -m pytest -v -rsx melodies_monet
 
       - name: Run docs/examples notebooks as scripts
         run: |
           cd docs/examples
           for f in *.ipynb; do
             if [ "$f" == 'idealized.ipynb' ]; then
-              jupytext --to py $f -o t.py
-              python t.py
+              jupytext --to py $f -o t.py && python t.py || exit 1
             fi
           done
           cd -
 
+      - name: Prepare idealized save/read cases
+        shell: python
+        run: |
+          from copy import deepcopy
+          import yaml
+
+          with open('docs/examples/control_idealized.yaml') as f:
+            ctl = yaml.safe_load(f)
+          assert {'save', 'read'} < ctl['analysis'].keys()
+
+          ctl_save = deepcopy(ctl)
+          del ctl_save['analysis']['read']
+          with open('docs/examples/control_idealized_save.yaml', 'w') as f:
+            yaml.safe_dump(ctl_save, f)
+
+          ctl_read = deepcopy(ctl)
+          del ctl_read['analysis']['save']
+          with open('docs/examples/control_idealized_read.yaml', 'w') as f:
+            yaml.safe_dump(ctl_read, f)
+
       - name: Check CLI works
         run: |
           cd docs/examples
           melodies-monet --version
           python -m melodies_monet --version
-          melodies-monet run control_idealized.yaml
+          melodies-monet run control_idealized_save.yaml
+          melodies-monet run control_idealized_read.yaml
           cd -
 
   docs:
@@ -78,13 +95,13 @@ jobs:
         shell: bash -l {0}
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v12
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
+          cache-environment: true
 
       - name: linkcheck
         run: sphinx-build -b linkcheck docs docs/_build/linkcheck
@@ -107,7 +124,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Check that .py files have the license header
         run: python3 ci/check-for-license-header.py

docs/appendix/machine-specific-install.rst

@@ -1,11 +1,11 @@
 Machine-specific Install
 ========================
 
-NCAR HPC Cheyenne/Casper
+NCAR HPC Derecho/Casper
 ------------------------
 
 Below are specific recipes for getting started with MELODIES MONET
-on the NCAR HPC, Cheyenne/Casper.
+on the NCAR HPC, Derecho/Casper.
 
 **Official NCAR JupyterHub kernel**
 
@@ -60,7 +60,7 @@ environment for running and developing MELODIES MONET.
    * You will need a NOAA HPC account to access the RDHPCS wiki link above.
 
    * Both Anaconda/Miniconda will work well for MELODIES MONET. See
-     `conda instructions <https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda>`__
+     `conda instructions <https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html#installing-conda>`__
      to determine, which is the best option for you.
 
    * Pick a directory for your download and run the following wget command with 

docs/appendix/troubleshooting.rst

@@ -0,0 +1,18 @@
+Troubleshooting
+===============
+
+Installation problems
+---------------------
+* Conda installation fails:
+    * Often the problem is in the installation of wrf-python. Check that your computer does not have an Apple Silicon CPU (Apple Intel should be fine) and that the Python version is compatible with the wrf-python Conda package.
+
+Runtime issues
+--------------
+* Pairing (:meth:`~melodies_monet.driver.analysis.pair_models`) takes too long:
+    * ``analysis.pair_models()`` is one of the most expensive functions in MELODIES MONET, and you might be running out of memory. If you have access to more RAM, try it with that. A Time Chunking functionality is being developed to deal with this issue. Check :doc:`/users_guide/time_chunking`.
+* The plots are not produced. The error message references ``leg.legendHandles``:
+    * You are probably using Matplotlib 3.9+ with pandas 1.x. Downgrade Matplotlib to 3.8 (upgrading pandas should also work, but you might run into some incompatibilities for some specific functionalities, especially those related to downloading observational data with MONETIO. Check :doc:`/getting_started/installation`).
+* NaN does not exist, or pandas not detecting NaN values, etc:
+    * In NumPy 2.x, ``np.NaN`` has been completely replaced by ``np.nan``. ``np.NaN`` is required by some versions of MONETIO. There are two solutions: (i) downgrading NumPy to previous versions (``'numpy<2'``) or (ii) upgrading MONETIO (``'monetio>=0.2.7'``). If you go with option (ii), you might need to use the _develop_ branch of MONETIO (this should make it into _stable_ soon).
+* Failure downoloading maps:
+    * MELODIES-MONET uses Cartopy for mapping. Cartopy downloads shapefiles automatically to create the maps, and if you are offline or working on a machine with download restrictions, this might fail. Check :doc:`/appendix/machine-specific-install`. This is the situation for *NOAA HPC Hera*, and the solution discussed there should work.

docs/appendix/yaml.rst

@@ -106,6 +106,10 @@ data (e.g., surf_only: True).
 Typically this is set at the horizontal resolution of your model * 1.5. Setting 
 this to a smaller value will speed up the pairing process. 
 
+**apply_ak:** This is an optional argument used for pairing of satellite data. This
+should be set to True when application of satellite averaging kernels or apriori data 
+to model observations is desired. 
+
 **mapping:** This is the mapping dictionary for all variables to be plotted. 
 For each observational dataset, add a mapping dictionary where the model 
 variable name is first (i.e., key) and the observation variable name is second 
@@ -181,7 +185,7 @@ Generalizing this to include other surface observations is under development.
 **filename:**  The file directory location and name. These observations need 
 to be preprocessed prior to incorporating them into MELODIES MONET.
 Shell variables prefixed with the ``$`` symbol, such as ``$HOME``, will be expanded.
-See :doc:`../tutorial/downloading_obs` for more details.
+See :doc:`../getting_started/downloading_obs` for more details.
 
 **obs_type:** The observation type. Options are: "pt_sfc" or point surface. Adding 
 options for Aircraft and Satellite observations are under development.
@@ -227,7 +231,7 @@ nan_values are set to NaN first and then the unit conversion is applied.
      (in %) is used to calculate the percentile (e.g., 5, 50, 95). Currently only
      used for "spatial_bias" plots. Will work with data as is and regulatory metrics.
    * **regulatory:** If false (default), use data as is. If set to true, the
-     regulatory metric is calculated as explained under :doc:`/background/supported_analyses`.
+     regulatory metric is calculated as explained under :doc:`/users_guide/supported_diagnostics`.
      Only works for "OZONE" and "PM2.5" variables.
    * **ylabel_reg_plot:** String to use as ylabel in plot for regulatory calculation.
      Useful for adding units or instrument information. Only used if regulatory = True.
@@ -272,15 +276,15 @@ Plots
 -----
 All input for each plotting group. A plotting group consists of one plotting 
 type. The plotting types are described in 
-:doc:`/background/supported_plots`. All model /
+:doc:`/users_guide/supported_plots`. All model /
 observational pairs and domains specified for the plotting group will be 
 included. You may include as many plotting groups as you like.
 
 For each plotting group, update the label and include the following information.
 Note: the labels need to be unique, but otherwise are not used.
 
 **type:** The plot type. Options are: "timeseries", "taylor", "spatial_bias",
-"spatial_overlay", "spatial_bias_exceedance", and "boxplot"
+"spatial_overlay", "spatial_bias_exceedance", "boxplot", "multi-boxplot","csi"
 Note: "spatial_bias_exceedance" plots only work when regulatory = True.
 
 **fig_kwargs:** This is optional to provide a dictionary with figure 
@@ -313,12 +317,40 @@ For example, ::
 **domain_type:** List of domain types to be plotted. These correspond with
 the columns in the observation file. (e.g., airnow: epa_region, state_name, 
 siteid, etc.).
+For automatic EPA or Giorgi region boxes (if they are not included
+with the columns in the observation file), choose ``auto-region:epa`` or
+``auto-region:giorgi``. Take into account that ``auto-region:epa`` is only a rough
+approximation, since it assumes perfect, rectangular lonlat boxes.
 
 **domain_name:** List of domain names to be plotted. If domain_type = all, all 
 data will be used and the domain_name is used only in the plot title. If 
 domain_type is not equal to all, MELODIES MONET will query all of the data 
 where domain_type is equal to domain_name.
 
+**region_name:** list of source of regions used in title.
+(e.g., ['epa_region'])
+
+**region_list:** list of regions we will calculate for scorecard. 
+(e.g., ['R1','R2','R3','R4','R5','R6','R7','R8','R9','R10']
+
+**urban_rural_name:** list of only one string input, which is variable used to
+determine wheter urban or rural site. (e.g., ['msa_name'])
+
+**urban_rural_differentiate_value:** string of value used to determine whether 
+variable is rural or urban. (e.g., '').
+
+**better_or_worse_method:** string of method used to determine which models 
+is better compared to observations. (e.g., 'RMSE', 'IOA' ,' NMB', 'NME'). choose
+one only for each time scorecard code run.
+
+**model_name_list:** 
+for multi-box plot, list of observation and model names user choose to set as x-labels; 
+for csi plot, list of model names (only) user choose to set as labels.
+
+**threshold_list:** csi plot only. list of values used as x variables. example: [10,20,30,40,50,60,70,80,90,100] 
+
+**score_name:** csi plot only. list of scores user can choose to plot. examples are "Critical Success Index' 'False Alarm Rate' 'Hit Rate'.
+
 **data:** This a list of model / observation pairs to be plotted where the 
 observation label is first and the model label is second 
 (e.g., ['airnow_cmaq_expt', 'airnow_rrfs_13km', 'airnow_wrfchem_v4.2'])
@@ -362,14 +394,14 @@ observation label is first and the model label is second
      used for averaging and plotting. Options are 'time' for UTC or 'time_local' 
      for local time
    * **ts_avg_window:** This is for timeseries plots only. This is the averaging 
-     window applied to the data. Options are None for no averaging or a pandas 
-     resample rule (e.g., 'H' is hourly, 'D' is daily).
+     window applied to the data. No averaging done if not provided in the yaml file (i.e., ts_avg_window is optional). Averaging is done if a pandas 
+     resample rule (e.g., 'H' is hourly, 'D' is daily) is specified.
 
 Stats
 -----
 All input needed to calculate the statistics. The supported statistics available 
 in MELODIES MONET are described in 
-:doc:`/background/supported_stats`. All model /
+:doc:`/users_guide/supported_stats`. All model /
 observational pairs and domains specified will be included. You may include as 
 many statistics as you like. Note however that the calculation of the statistics 
 is relatively slow right now. Optimizing this code is under development.
@@ -379,7 +411,7 @@ use Kelvin. Wind direction has special calculations for AirNow if the observatio
 name is 'WD'. 
 
 **stat_list:** List of acronyms of statistics to calculate as defined in 
-:doc:`/background/supported_stats`. (e.g., ['MB', 'MdnB',
+:doc:`/users_guide/supported_stats`. (e.g., ['MB', 'MdnB',
 'NMB', 'NMdnB','R2', 'RMSE']). A dictionary of definitions is also included in 
 MELODIES-MONET/melodies_monet/stats/proc_stats.py. 
 

docs/cli.rst

@@ -15,6 +15,9 @@ any Python code::
 * |run|_ -- run a control file
 * |get-airnow|_ -- get AirNow data
 * |get-aeronet|_ -- get AERONET data
+* |get-aqs|_ -- get AQS data
+* |get-ish|_ -- get ISH data
+* |get-ish-lite|_ -- get ISH-Lite data
 
 .. |run| replace:: ``run``
 .. _run: #melodies-monet-run
@@ -25,6 +28,14 @@ any Python code::
 .. |get-aeronet| replace:: ``get-aeronet``
 .. _get-aeronet: #melodies-monet-get-aeronet
 
+.. |get-aqs| replace:: ``get-aqs``
+.. _get-aqs: #melodies-monet-get-aqs
+
+.. |get-ish| replace:: ``get-ish``
+.. _get-ish: #melodies-monet-get-ish
+
+.. |get-ish-lite| replace:: ``get-ish-lite``
+.. _get-ish-lite: #melodies-monet-get-ish-lite
 
 .. click:: melodies_monet._cli:_typer_click_object
    :prog: melodies-monet

docs/conf.py

@@ -40,8 +40,8 @@
 ]
 
 extlinks = {
-    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH'),
-    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR'),
+    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH %s'),
+    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR %s'),
 }
 
 autosummary_generate = True  # default in Sphinx v4
@@ -88,7 +88,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -198,7 +198,10 @@
     "https://www2.cisl.ucar.edu/resources/conda-environments",
     # Sphinx 4.5 linkcheck having problem:
     "https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account",
+    # NCEI sites having problems
+    "https://www.ncdc.noaa.gov/crn/",
 ]
+user_agent = "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
 
 autosectionlabel_prefix_document = True
 autosectionlabel_maxdepth = 2

docs/develop/contribute.rst

@@ -11,8 +11,8 @@ the contributions and support from you.
 - Please check out 
   `GitHub Projects <https://github.com/NOAA-CSL/MELODIES-MONET/projects>`__ 
   to learn about current development plans.
-- Join an :ref:`Upcoming Workshop <develop/workshops:Upcoming>`
-  or check out :ref:`Past Workshops <develop/workshops:Past>`.
+- Join an :ref:`Upcoming Workshop <develop/other_resources:Upcoming>`
+  or check out :ref:`Past Workshops <develop/other_resources:Past>`.
 - See our developers guide, to learn 
   :ref:`develop/developers_guide:How to incorporate updates to MELODIES MONET`.
 - Email melodies@ucar.edu with questions.

docs/develop/datasets.rst

@@ -21,3 +21,58 @@ While a part of the MONETIO repository,
 the private MELODIES MONET readers are designated with prefix ``_mm``.
 
 Support for additional models is also under developed.
+
+Standard variables are required to be computed in each model reader for each capability including surface, aircraft, and satellite as specified in the table below.
+
+.. list-table:: Required Variables for Model Readers
+   :widths: 10 30 30 30
+   :header-rows: 1
+
+   * - Capability
+     - | Variable Name 
+       | in Code
+     - Description
+     - Additional Requirements
+   * - Surface
+     - | ``time``
+       | ``latitude``
+       | ``longitude``
+     - | Time in ``datetime64[ns]`` format
+       | Latitude in degrees
+       | Longitude in degrees
+     - | Provide only surface model data 
+       | or if provide vertical model data, 
+       | first level must be the level 
+       | nearest to the surface.
+       | All gases are in ppb and 
+       | all aerosols are in µg/m3.
+   * - Aircraft
+     - | ``time``
+       | ``latitude``
+       | ``longitude``
+       | ``pres_pa_mid``
+       | ``temperature_k``
+     - | Time in ``datetime64[ns]`` format
+       | Latitude in degrees
+       | Longitude in degrees
+       | Mid-level pressure in pascals (Pa)
+       | Mid-level temperature in kelvin (K)
+     - | Provide vertical model data. 
+       | All gases are in ppb and 
+       | all aerosols are in µg/m3.
+   * - Satellites
+     - | ``time``
+       | ``latitude``
+       | ``longitude``
+       | ``pres_pa_mid``
+       | ``temperature_k``
+       | ``dz_m``
+       | ``surfpres_pa``
+     - | Time in ``datetime64[ns]`` format
+       | Latitude in degrees
+       | Longitude in degrees
+       | Mid-level pressure in pascals (Pa)
+       | Mid-level temperature in kelvin (K)
+       | Layer thickness in meters (m)
+       | Surface pressure in pascals (Pa)
+     - | Provide vertical model data.