Moved to users_guide subdir.

.github/workflows/ci.yml

@@ -17,17 +17,17 @@ jobs:
     strategy:
       matrix:
         monet: [cf, dev]
-      fail-fast: false  # just until cf versions are working
+      fail-fast: false  # always both
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v15
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
-          extra-specs: |
+          cache-environment: true
+          create-args: >-
             pytest
 
       - if: ${{ matrix.monet == 'dev' }}
@@ -53,8 +53,7 @@ jobs:
           cd docs/examples
           for f in *.ipynb; do
             if [ "$f" == 'idealized.ipynb' ]; then
-              jupytext --to py $f -o t.py
-              python t.py
+              jupytext --to py $f -o t.py && python t.py || exit 1
             fi
           done
           cd -
@@ -96,13 +95,13 @@ jobs:
         shell: bash -l {0}
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v15
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
+          cache-environment: true
 
       - name: linkcheck
         run: sphinx-build -b linkcheck docs docs/_build/linkcheck
@@ -125,7 +124,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Check that .py files have the license header
         run: python3 ci/check-for-license-header.py

docs/appendix/machine-specific-install.rst

@@ -1,11 +1,11 @@
 Machine-specific Install
 ========================
 
-NCAR HPC Cheyenne/Casper
+NCAR HPC Derecho/Casper
 ------------------------
 
 Below are specific recipes for getting started with MELODIES MONET
-on the NCAR HPC, Cheyenne/Casper.
+on the NCAR HPC, Derecho/Casper.
 
 **Official NCAR JupyterHub kernel**
 

docs/appendix/troubleshooting.rst

@@ -0,0 +1,18 @@
+Troubleshooting
+===============
+
+Installation problems
+---------------------
+* Conda installation fails:
+    * Often the problem is in the installation of wrf-python. Check that your computer does not have an Apple Silicon CPU (Apple Intel should be fine) and that the Python version is compatible with the wrf-python Conda package.
+
+Runtime issues
+--------------
+* Pairing (:meth:`~melodies_monet.driver.analysis.pair_models`) takes too long:
+    * ``analysis.pair_models()`` is one of the most expensive functions in MELODIES MONET, and you might be running out of memory. If you have access to more RAM, try it with that. A Time Chunking functionality is being developed to deal with this issue. Check :doc:`/users_guide/time_chunking`.
+* The plots are not produced. The error message references ``leg.legendHandles``:
+    * You are probably using Matplotlib 3.9+ with pandas 1.x. Downgrade Matplotlib to 3.8 (upgrading pandas should also work, but you might run into some incompatibilities for some specific functionalities, especially those related to downloading observational data with MONETIO. Check :doc:`/getting_started/installation`).
+* NaN does not exist, or pandas not detecting NaN values, etc:
+    * In NumPy 2.x, ``np.NaN`` has been completely replaced by ``np.nan``. ``np.NaN`` is required by some versions of MONETIO. There are two solutions: (i) downgrading NumPy to previous versions (``'numpy<2'``) or (ii) upgrading MONETIO (``'monetio>=0.2.7'``). If you go with option (ii), you might need to use the _develop_ branch of MONETIO (this should make it into _stable_ soon).
+* Failure downoloading maps:
+    * MELODIES-MONET uses Cartopy for mapping. Cartopy downloads shapefiles automatically to create the maps, and if you are offline or working on a machine with download restrictions, this might fail. Check :doc:`/appendix/machine-specific-install`. This is the situation for *NOAA HPC Hera*, and the solution discussed there should work.

docs/appendix/yaml.rst

@@ -185,7 +185,7 @@ Generalizing this to include other surface observations is under development.
 **filename:**  The file directory location and name. These observations need 
 to be preprocessed prior to incorporating them into MELODIES MONET.
 Shell variables prefixed with the ``$`` symbol, such as ``$HOME``, will be expanded.
-See :doc:`../tutorial/downloading_obs` for more details.
+See :doc:`../getting_started/downloading_obs` for more details.
 
 **obs_type:** The observation type. Options are: "pt_sfc" or point surface. Adding 
 options for Aircraft and Satellite observations are under development.
@@ -231,7 +231,7 @@ nan_values are set to NaN first and then the unit conversion is applied.
      (in %) is used to calculate the percentile (e.g., 5, 50, 95). Currently only
      used for "spatial_bias" plots. Will work with data as is and regulatory metrics.
    * **regulatory:** If false (default), use data as is. If set to true, the
-     regulatory metric is calculated as explained under :doc:`/background/supported_analyses`.
+     regulatory metric is calculated as explained under :doc:`/users_guide/supported_diagnostics`.
      Only works for "OZONE" and "PM2.5" variables.
    * **ylabel_reg_plot:** String to use as ylabel in plot for regulatory calculation.
      Useful for adding units or instrument information. Only used if regulatory = True.
@@ -276,7 +276,7 @@ Plots
 -----
 All input for each plotting group. A plotting group consists of one plotting 
 type. The plotting types are described in 
-:doc:`/background/supported_plots`. All model /
+:doc:`/users_guide/supported_plots`. All model /
 observational pairs and domains specified for the plotting group will be 
 included. You may include as many plotting groups as you like.
 
@@ -397,7 +397,7 @@ Stats
 -----
 All input needed to calculate the statistics. The supported statistics available 
 in MELODIES MONET are described in 
-:doc:`/background/supported_stats`. All model /
+:doc:`/users_guide/supported_stats`. All model /
 observational pairs and domains specified will be included. You may include as 
 many statistics as you like. Note however that the calculation of the statistics 
 is relatively slow right now. Optimizing this code is under development.
@@ -407,7 +407,7 @@ use Kelvin. Wind direction has special calculations for AirNow if the observatio
 name is 'WD'. 
 
 **stat_list:** List of acronyms of statistics to calculate as defined in 
-:doc:`/background/supported_stats`. (e.g., ['MB', 'MdnB',
+:doc:`/users_guide/supported_stats`. (e.g., ['MB', 'MdnB',
 'NMB', 'NMdnB','R2', 'RMSE']). A dictionary of definitions is also included in 
 MELODIES-MONET/melodies_monet/stats/proc_stats.py. 
 

docs/conf.py

@@ -40,8 +40,8 @@
 ]
 
 extlinks = {
-    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH'),
-    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR'),
+    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH %s'),
+    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR %s'),
 }
 
 autosummary_generate = True  # default in Sphinx v4
@@ -88,7 +88,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.

docs/develop/contribute.rst

@@ -11,8 +11,8 @@ the contributions and support from you.
 - Please check out 
   `GitHub Projects <https://github.com/NOAA-CSL/MELODIES-MONET/projects>`__ 
   to learn about current development plans.
-- Join an :ref:`Upcoming Workshop <develop/workshops:Upcoming>`
-  or check out :ref:`Past Workshops <develop/workshops:Past>`.
+- Join an :ref:`Upcoming Workshop <develop/other_resources:Upcoming>`
+  or check out :ref:`Past Workshops <develop/other_resources:Past>`.
 - See our developers guide, to learn 
   :ref:`develop/developers_guide:How to incorporate updates to MELODIES MONET`.
 - Email melodies@ucar.edu with questions.

docs/develop/developers_guide.rst

@@ -55,10 +55,10 @@ In order to prepare for developing MELODIES MONET,
 we clone the repositories and create a conda environment that references them.
 
 .. note::
-   If you are installing MELODIES MONET on NCAR Cheyenne or NOAA Hera
+   If you are installing MELODIES MONET on NCAR Casper or NOAA Hera
    follow these machine specific instructions instead.
 
-   - :ref:`NCAR Cheyenne <appendix/machine-specific-install:NCAR HPC Cheyenne/Casper>`
+   - :ref:`NCAR Casper <appendix/machine-specific-install:NCAR HPC Derecho/Casper>`
    - :ref:`NOAA Hera <appendix/machine-specific-install:NOAA HPC Hera>`
 
 .. important::
@@ -74,7 +74,7 @@ these instructions:
 
        $ conda create --name melodies-monet python=3.9
        $ conda activate melodies-monet
-       $ conda install -y -c conda-forge pyyaml pandas=1 monet monetio netcdf4 wrf-python typer rich pooch jupyterlab
+       $ conda install -y -c conda-forge pyyaml pandas=1 'matplotlib-base<3.9' monet monetio netcdf4 wrf-python typer rich pooch jupyterlab
 
 (b) Clone [#clone]_ and link the latest development versions of MONET and MONETIO from GitHub to
     your conda environment::
@@ -140,8 +140,8 @@ changes.
     changes. 
 
 
-User's Guide Development
-------------------------
+Contributions to the Docs
+-------------------------
 
 If you add a component to MELODIES MONET, please follow the instructions below 
 to update the readthedocs user guide. 
@@ -172,7 +172,7 @@ The generated HTML will be created in ``docs/_build/html``,
 with ``docs/_build/html/index.html`` the main page that can be
 viewed in any browser.
 
-Please see the `Documentation <https://github.com/NOAA-CSL/MELODIES-MONET/projects/2>`_ 
+Please see the `MELODIES-MONET Documentation <https://github.com/orgs/NOAA-CSL/projects/6>`_ 
 project on GitHub to learn about current and future development.
 
 

docs/develop/workshops.rst → docs/develop/other_resources.rst

@@ -1,5 +1,5 @@
-Tutorials & Workshops
-=====================
+Other Resources
+===============
 
 Check out our upcoming and past tutorials, workshops, surveys, etc. below.
 

docs/environment-docs.yml

@@ -7,6 +7,7 @@ dependencies:
   - python=3.9
   #
   # melodies_monet deps
+  - matplotlib-base<3.9  # for pandas v1 compat
   - monet
   - monetio
   - netcdf4

docs/examples/control_camchem.yaml

@@ -86,7 +86,7 @@ plots:
     data_proc:
       rem_obs_nan: True # True: Remove all points where model or obs variable is NaN. False: Remove only points where model variable is NaN.
       ts_select_time: 'time' # Time used for avg and plotting: Options: 'time' for UTC or 'time_local'
-      ts_avg_window: 'H' # Options: None for no averaging or list pandas resample rule (e.g., 'H', 'D')
+      ts_avg_window: 'h' # Options: None for no averaging or list pandas resample rule (e.g., 'h', 'D')
       set_axis: False # If select True, add vmin_plot and vmax_plot for each variable in obs.
   plot_grp2:
     type: 'taylor' # plot type

docs/examples/control_camchem_se.yaml

@@ -86,7 +86,7 @@ plots:
     data_proc:
       rem_obs_nan: True # True: Remove all points where model or obs variable is NaN. False: Remove only points where model variable is NaN.
       ts_select_time: 'time_local' #Time used for avg and plotting: Options: 'time' for UTC or 'time_local'
-      ts_avg_window: 'H' # Options: None for no averaging or list pandas resample rule (e.g., 'H', 'D')
+      ts_avg_window: 'h' # Options: None for no averaging or list pandas resample rule (e.g., 'h', 'D')
       set_axis: False #If select True, add vmin_plot and vmax_plot for each variable in obs.
   plot_grp2:
     type: 'taylor' # plot type

docs/examples/control_idealized.yaml

@@ -61,7 +61,7 @@ plots:
     data_proc:  # These four seem to be required for time series
       rem_obs_nan: True  # True: Remove all points where model or obs variable is NaN. False: Remove only points where model variable is NaN.
       ts_select_time: 'time'  # 'time' for UTC or 'time_local'
-      ts_avg_window: '3H'  # Options: None for no averaging or list pandas resample rule (e.g., 'H', 'D')
+      ts_avg_window: '3h'  # Options: None for no averaging or list pandas resample rule (e.g., 'h', 'D')
       # ^ TODO: null setting seems not working
       set_axis: False  # If True, add vmin_plot and vmax_plot for each variable in obs.
 

docs/examples/control_wrfchem_mech-0905_2.yaml

@@ -126,8 +126,8 @@ plots:
       ts_select_time: "time_local"  # `ts_` indicates this is time series plot-specific
       # ^ Time used for avg and plotting
       #   Options: 'time' for UTC or 'time_local'
-      ts_avg_window: "H"
-      # ^ Options: None for no averaging, pandas resample rule (e.g., 'H', 'D')
+      ts_avg_window: "h"
+      # ^ Options: None for no averaging, pandas resample rule (e.g., 'h', 'D')
       set_axis: True
       # ^ If true, add `vmin_plot` and `vmax_plot` for each variable in obs.
 

docs/examples/control_wrfchem_saveandread.yaml

@@ -147,8 +147,8 @@ plots:
       ts_select_time: "time_local"  # `ts_` indicates this is time series plot-specific
       # ^ Time used for avg and plotting
       #   Options: 'time' for UTC or 'time_local'
-      ts_avg_window: "H"
-      # ^ Options: None for no averaging, pandas resample rule (e.g., 'H', 'D')
+      ts_avg_window: "h"
+      # ^ Options: None for no averaging, pandas resample rule (e.g., 'h', 'D')
       set_axis: True
       # ^ If true, add `vmin_plot` and `vmax_plot` for each variable in obs.
 

docs/tutorial/downloading_obs.rst → docs/getting_started/downloading_obs.rst

@@ -9,9 +9,18 @@ so that users do not have to re-download observational data over and over again
 for the same analysis period. We will work on automating this process further 
 in the future. 
 
-Examples of preprocessed observational data for MELODIES MONET are here:
+
+Note: for users using MELODIES MONET on the NOAA Hera machine (or other machines 
+with download restrictions), you will need to run these jupyter notebooks on a 
+machine without download restrictions and manually copy the netCDF files produced 
+onto the NOAA Hera machine.
+
+Examples of preprocessed surface observational data for MELODIES MONET are here:
 `MELODIES MONET Example Datasets <https://csl.noaa.gov/groups/csl4/modeldata/melodies-monet/>`_. 
 
+Surface
+-------
+
 In order to preprocess the observational data for additional time periods 
 follow the instructions in the jupyter notebooks in the 
 ``examples/process_obs`` folder of the code on GitHub. Examples for 
@@ -24,11 +33,12 @@ the following observational datasets are provided.
 Adapting these scripts for other observational datasets should be straight 
 forward.
 
-Note: for users using MELODIES MONET on the NOAA Hera machine (or other machines 
-with download restrictions), you will need to run these jupyter notebooks on a 
-machine without download restrictions and manually copy the netCDF files produced 
-onto the NOAA Hera machine.
-
+Aircraft
+--------
 
+Under development. Some functionality is alrady available, and can be looked at in the official `Github repo routine <https://github.com/NOAA-CSL/MELODIES-MONET/blob/develop/melodies_monet/plots/aircraftplots.py/>`__.
 
+Satellite
+---------
 
+Under development. Some functionality is already available, and can be looked at  in the official `Github repo routine <https://github.com/NOAA-CSL/MELODIES-MONET/blob/develop/melodies_monet/plots/satplots.py/>`__.

docs/tutorial/how_to_run.rst → docs/getting_started/how_to_run.rst

@@ -40,13 +40,13 @@ structure is the following:
 
 * **plots** -- All input for each plotting group. A plotting group consists 
   of one plotting type. The plotting types are described in 
-  :doc:`/background/supported_plots`. All model /
+  :doc:`/users_guide/supported_plots`. All model /
   observational pairs and domains specified for the plotting group will be 
   included. You may include as many plotting groups as you like.
 
 * **stats** -- All input needed to calculate the statistics. The supported
   statistics available in MELODIES MONET are described in 
-  :doc:`/background/supported_stats`. All model /
+  :doc:`/users_guide/supported_stats`. All model /
   observational pairs and domains specified will be included. You may 
   include as many statistics as you like. Note however that the calculation 
   of the statistics is relatively slow right now. Optimizing this code is 

docs/tutorial/installation.rst → docs/getting_started/installation.rst

@@ -4,7 +4,8 @@ Installation/Requirements
 Required dependencies [#yaml]_
 ------------------------------
 
-- Python 3.6+ (3.9 recommended)
+- Python 3.6+ (3.9 -- 3.11 recommended)
+- ``pandas`` 1 (most of the system works with pandas 2 as well, but some of the :doc:`/cli` commands for downloading observational datasets using MONETIO might fail)
 - ``pyyaml`` (to read control files)
 - ``monet``, which brings `many dependencies <https://monet-arl.readthedocs.io/en/stable/installing.html>`__
 - ``monetio``, which brings `a few dependencies <https://monetio.readthedocs.io/en/stable/installing.html>`__
@@ -13,11 +14,16 @@ Optional dependencies
 ---------------------
 
 - ``netcdf4`` (`from Unidata <https://unidata.github.io/netcdf4-python/>`__; most likely needed for reading model/obs datasets)
-- ``wrf-python`` (needed in order to use the WRF-Chem reader)
+- ``wrf-python`` (needed in order to use the WRF-Chem reader, currently the conda package seems to require Python < 3.12)
 - ``typer`` (to use the :doc:`/cli`;
   add ``rich`` `for <https://typer.tiangolo.com/release-notes/#060>`__ fancy tracebacks and ``--help``)
 - ``pooch`` (to enable automatic downloading of :doc:`tutorial datasets </examples/tutorial-data>`)
 
+Incompatibilities
+-----------------
+- pandas=1 is incompatible with matplotlib 3.9+.
+- wrf-python, at least in the official conda-forge package, is not available for Python 3.12+, until `this build issue <https://github.com/conda-forge/wrf-python-feedstock/pull/70>`__ is resolved.
+
 .. _user-install-instructions:
 
 General instructions
@@ -34,8 +40,11 @@ First create and activate a conda environment::
 
 Add dependencies from conda-forge::
 
-    $ conda install -y -c conda-forge pyyaml pandas=1 monet monetio netcdf4 wrf-python typer rich pooch
+    $ conda install -y -c conda-forge pyyaml pandas=1 'matplotlib-base<3.9' monet monetio netcdf4 wrf-python typer rich pooch
 
+.. note::
+   Currently, the wrf-python conda package is not compatible with Apple Silicon (Apple machines using Intel should be fine). If you need to run the WRF-Chem reader and only have access to a machine using Apple Silicon, you can try compiling it from souce code from the official repos.
+
 Now, install the stable branch of MELODIES MONET to the environment::
 
     $ pip install --no-deps https://github.com/NOAA-CSL/MELODIES-MONET/archive/main.zip
@@ -51,5 +60,5 @@ Now, install the stable branch of MELODIES MONET to the environment::
    that include a record
    of all the dependencies (direct and indirect) are available via the GitHub:
 
-   - `NCAR Cheyenne environment.yml <https://github.com/NOAA-CSL/MELODIES-MONET/tree/develop/python_env_ymls/cheyenne>`__
+   - `NCAR Casper environment.yml <https://github.com/NOAA-CSL/MELODIES-MONET/tree/develop/python_env_ymls/casper>`__
    - `NOAA Hera environment.yml <https://github.com/NOAA-CSL/MELODIES-MONET/tree/develop/python_env_ymls/hera>`__

docs/tutorial/getting_started.rst → docs/getting_started/software_architecture.rst

@@ -1,5 +1,5 @@
-Getting Started
-===============
+Software Architecture
+=====================
 
 The User Interface
 ------------------
@@ -65,9 +65,9 @@ we aim for the average user of MELODIES MONET to be able to easily understand
 the core code and contribute new components.
 
 To learn more about current capabilities and future development objectives
-see the :doc:`/background/supported_datasets` ,
-:doc:`/background/supported_plots`, and
-:doc:`/background/supported_stats` pages in the
+see the :doc:`/users_guide/supported_datasets` ,
+:doc:`/users_guide/supported_plots`, and
+:doc:`/users_guide/supported_stats` pages in the
 Background section of this guide.