Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New docker build for ravenpy birdy xclim #75

Merged
merged 14 commits into from
Jun 15, 2021
Merged

New docker build for ravenpy birdy xclim #75

merged 14 commits into from
Jun 15, 2021

Conversation

tlvu
Copy link
Contributor

@tlvu tlvu commented May 27, 2021 •
edited
Loading

Overview

New Jupyter env to get latest of everything, including ravenpy, birdy, xclim.

Jenkins build passing with updated Finch and Pavics-sdi notebooks: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/25/

Matching Finch notebook PR: bird-house/finch#188
Matching Pavics-sdi notebook PR: Ouranosinc/pavics-sdi#222
Matching PR to deploy this Jupyter env to PAVICS: bird-house/birdhouse-deploy#175

Matching Raven notebook failure found Ouranosinc/raven#392 (http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/14/console) and matching fix PR Ouranosinc/raven#393 (http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/24/).

Also added new libraries for upcoming Raven notebooks (fixes Ouranosinc/raven#394).

Pavics-landing homepage notebooks manually tested by @tlogan2000

Deployed to staging env for manual testing: https://medus.ouranos.ca/jupyter/

Also added a way to quickly test a new package without having to do a full rebuild.

Changes

<   - ravenpy=0.4.2=py37_1
>   - ravenpy=0.5.2=pyh7f9bfb9_0

# Renamed.
<   - raven=3.0.4.318=hc9bffa2_2
>   - raven-hydro=3.0.4.322=h516393e_0

<   - ostrich=21.03.16=h2bc3f7f_0
>   - ostrich=21.03.16=h4bd325d_1

<   - xclim=0.25.0=pyhd8ed1ab_0
>   - xclim=0.27.0=pyhd8ed1ab_0

# Old version was from pip.
<     - birdhouse-birdy==0.7.0
>   - birdy=v0.8.0=pyh6c4a22f_0

# Was previously included in another package, now it is standalone.
>   - pydantic=1.8.2=py37h5e8e339_0

# New libs for upcoming Raven notebooks
>   - gcsfs=2021.6.0=pyhd8ed1ab_0
>   - intake=0.6.2=pyhd8ed1ab_0
>   - intake-esm=2021.1.15=pyhd8ed1ab_0
>   - zarr=2.8.3=pyhd8ed1ab_0

<   - xarray=0.17.0=pyhd8ed1ab_0
>   - xarray=0.18.2=pyhd8ed1ab_0

<   - owslib=0.23.0=pyhd8ed1ab_0
>   - owslib=0.24.1=pyhd8ed1ab_0

<   - cf_xarray=0.5.1=pyh44b312d_0
>   - cf_xarray=0.5.2=pyh6c4a22f_0

<   - clisops=0.6.3=pyh44b312d_0
>   - clisops=0.6.5=pyh6c4a22f_0

<   - dask=2021.2.0=pyhd8ed1ab_0
>   - dask=2021.6.0=pyhd8ed1ab_0

# Downgrade !
<   - gdal=3.2.1=py37hc5bc4e4_7
>   - gdal=3.1.4=py37h2ec2946_8

# Downgrade !
<   - rasterio=1.2.2=py37hd5c4cce_0
>   - rasterio=1.2.1=py37ha549118_0

<   - hvplot=0.7.1=pyh44b312d_0
>   - hvplot=0.7.2=pyh6c4a22f_0

<   - rioxarray=0.3.1=pyhd8ed1ab_0
>   - rioxarray=0.4.1.post0=pyhd8ed1ab_0

# Downgrade !
<   - xskillscore=0.0.19=pyhd8ed1ab_0                                                                                                                 
>   - xskillscore=0.0.18=py_1

Full diff of conda env export:
210415-210527.1-update210615-conda-env-export.diff.txt

Full new conda env export:
210527.1-update210615-conda-env-export.yml.txt

tlvu added 7 commits May 25, 2021 09:43
@tlvu
release: update to use image pavics/workflow-tests:210525
c4d6da1
@tlvu
docker: add new Dockerfile for testing new package
ad6d514 
Useful to avoid full rebuild just to test adding a new package or
upgrading an existing package.

Sort of a restore of commit cbfa5d5.
@tlvu
docker: add pydantic, was part of previous build but missing in curre…
8103b74 
…nt build

Previous build probably got pydantic via indirect dependency but somehow
this has changed so now have to add pydantic explicitly, for Raven notebooks.
@tlvu
docker: test latest RavenPy to fix Raven notebooks
757c44e 
The fix did not make it to RavenPy 0.5.1.

To fix error running Raven notebooks:
```
  ---------------------------------------------------------------------------
  ImportError                               Traceback (most recent call last)
  <ipython-input-1-021da278d129> in <module>
        2
        3 from birdy import WPSClient
  ----> 4 from ravenpy.utilities.testdata import get_file
        5 import os
        6 import xarray as xr

  /opt/conda/envs/birdy/lib/python3.7/site-packages/ravenpy/utilities/testdata.py in <module>
       14 from xarray import Dataset
       15 from xarray import open_dataset as _open_dataset
  ---> 16 from xarray.tutorial import file_md5_checksum
       17
       18 _default_cache_dir = Path.home() / ".raven_testing_data"

  ImportError: cannot import name 'file_md5_checksum' from 'xarray.tutorial' (/opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/tutorial.py)
```
@tlvu
release: update to use image pavics/workflow-tests:210526
155ff69
@tlvu
docker: remove conda channel zeitsperre, ravenpy now available from c…
eed7904 
…onda-forge
@tlvu
release: update to use image pavics/workflow-tests:210527
be6aa30
@tlvu tlvu requested review from huard, cjauvin and aulemahal May 27, 2021 16:31
tlvu added a commit to bird-house/birdhouse-deploy that referenced this pull request May 27, 2021
@tlvu
jupyter: new version for updated ravenpy, birdy and xclim
6a64c87 
See PR Ouranosinc/PAVICS-e2e-workflow-tests#75
for more details.

Relevant changes:
```diff
<   - raven=3.0.4.318=hc9bffa2_2
>   - raven-hydro=3.0.4.322=h516393e_0

<   - ravenpy=0.4.2=py37_1
>   - ravenpy=0.5.2=pyh7f9bfb9_0

<   - ostrich=21.03.16=h2bc3f7f_0
>   - ostrich=21.03.16=h4bd325d_1

<   - xclim=0.25.0=pyhd8ed1ab_0
>   - xclim=0.26.1=pyhd8ed1ab_0

<     - birdhouse-birdy==0.7.0
>   - birdy=v0.8.0=pyh6c4a22f_0

>   - pydantic=1.8.2=py37h5e8e339_0

<   - xarray=0.17.0=pyhd8ed1ab_0
>   - xarray=0.18.2=pyhd8ed1ab_0

<   - owslib=0.23.0=pyhd8ed1ab_0
>   - owslib=0.24.1=pyhd8ed1ab_0

<   - cf_xarray=0.5.1=pyh44b312d_0
>   - cf_xarray=0.5.2=pyh6c4a22f_0

<   - cftime=1.4.1=py37h902c9e0_0
>   - cftime=1.5.0=py37h6f94858_0

<   - clisops=0.6.3=pyh44b312d_0
>   - clisops=0.6.4=pyh6c4a22f_0

<   - dask=2021.2.0=pyhd8ed1ab_0
>   - dask=2021.5.0=pyhd8ed1ab_0

<   - gdal=3.2.1=py37hc5bc4e4_7
>   - gdal=3.1.4=py37h2ec2946_8

<   - hvplot=0.7.1=pyh44b312d_0
>   - hvplot=0.7.2=pyh6c4a22f_0

<   - rioxarray=0.3.1=pyhd8ed1ab_0
>   - rioxarray=0.4.1.post0=pyhd8ed1ab_0

<   - xskillscore=0.0.19=pyhd8ed1ab_0
>   - xskillscore=0.0.18=py_1
```

Full diff of `conda env export`:
[210415-210527-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6555480/210415-210527-conda-env-export.diff.txt)

Full new `conda env export`:
[210527-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6555483/210527-conda-env-export.yml.txt)
This was referenced May 27, 2021
Merged
Merged
Merged
@aulemahal
Copy link
Collaborator

The update to cftime==1.5.0 is supposed to break the notebooks of the PAVICS homepage, I gather they are not tested?

@tlvu
Copy link
Contributor Author

tlvu commented May 27, 2021

The update to cftime==1.5.0 is supposed to break the notebooks of the PAVICS homepage, I gather they are not tested?

@aulemahal
Nice catch, notebooks of PAVICS homepage are not part of regular test yet. It's a private git repo so had trouble hooking it into the standard test.

What would be a possible work-around, pin cftime to the previous version?

@tlogan2000 We have to have a plan to ensure testing of the PAVICS homepage notebooks. Each release of Jupyter env, I ask you to manually test them? Make the pavics-landing git repo public so it's simple to plug into the existing testing mechanism? Or I completely re-write the testing and tuto notebooks deployment to be able to handle private git repo?

@tlvu
Copy link
Contributor Author

tlvu commented May 27, 2021

This new Jupyter env is deployed to staging env for manual testing: https://medus.ouranos.ca/jupyter/

@aulemahal
Copy link
Collaborator

As sad as it is, I think pinning cftime is the current solution. In fact, the problem arises from nc-time-axis (see SciTools/nc-time-axis#58). That's the small plugin that allows us to plot with matplotlib when the time axis is made of cftime objects (instead of numpy objects).

@tlogan2000
Copy link
Collaborator

@tlvu I think we can make probably make pavics-landing public. Testing is definitely good idea but there are a couple of "advanced" sections that calculate a lot of data and take a while to run etc... we might need some precalculated data in the repo and a method to somehow skip a cell or two in a given notebook in the tests? We can try set up a plan for sure

@tlvu
Copy link
Contributor Author

tlvu commented May 28, 2021

New build with cftime pinning is on https://medus.ouranos.ca/jupyter/. Many build problems on DockerHub today.

@tlvu tlvu mentioned this pull request Jun 1, 2021
Closed
tlvu added a commit to Ouranosinc/raven that referenced this pull request Jun 10, 2021
@tlvu
Extract_geographical_watershed_properties.ipynb: fix run failure in n…
832d629 
…ew Jupyter env

Fixes #392.

With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below.

```
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 7: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  ValueError                                Traceback (most recent call last)
23:00:58  <ipython-input-8-8b167a3ecdc9> in <module>
23:00:58        4 features, statistics, grid = stats_resp.get(asobj=True)
23:00:58        5
23:00:58  ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58        7     with memfile.open() as dataset:
23:00:58        8         grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext)
23:00:58      103         """
23:00:58      104         super(MemoryFile, self).__init__(
23:00:58  --> 105             file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext)
23:00:58      106
23:00:58      107     @ensure_env
23:00:58
23:00:58  rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__()
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self)
23:00:58      127
23:00:58      128     def __bool__(self: Any) -> bool:
23:00:58  --> 129         return bool(self.values)
23:00:58      130
23:00:58      131     def __float__(self: Any) -> float:
23:00:58
23:00:58  ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 10: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58  display(unique, counts)
23:00:58
23:00:58  # Pixels values at '127' are NaN and can be ignored.
23:00:58  plt.bar(unique[:-1], counts[:-1])
23:00:58  plt.show()
23:00:58
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-11-a26c52126e9c> in <module>
23:00:58  ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58        2 display(unique, counts)
23:00:58        3
23:00:58        4 # Pixels values at '127' are NaN and can be ignored.
23:00:58        5 plt.bar(unique[:-1], counts[:-1])
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'sel'
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 11: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  import cartopy.crs as ccrs
23:00:58
23:00:58  # Set a CRS transformation:
23:00:58  crs = ccrs.LambertConformal(
23:00:58      central_latitude=49, central_longitude=-95, standard_parallels=(49, 77)
23:00:58  )
23:00:58
23:00:58  ax = plt.subplot(projection=crs)
23:00:58  grid.name = "Land Use Categories"
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-12-f9fdb6f0e562> in <module>
23:00:58        7
23:00:58        8 ax = plt.subplot(projection=crs)
23:00:58  ----> 9 grid.name = "Land Use Categories"
23:00:58       10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58       11 plt.show()
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'name'
```
@tlvu tlvu mentioned this pull request Jun 10, 2021
Merged
tlvu added a commit to bird-house/birdhouse-deploy that referenced this pull request Jun 15, 2021
@tlvu
jupyter: update to version 210527.1-update20210615
d2fa241 
Relevant changes:

```diff
<   - xclim=0.25.0=pyhd8ed1ab_0
>   - xclim=0.27.0=pyhd8ed1ab_0

>   - gcsfs=2021.6.0=pyhd8ed1ab_0
>   - intake=0.6.2=pyhd8ed1ab_0
>   - intake-esm=2021.1.15=pyhd8ed1ab_0
>   - zarr=2.8.3=pyhd8ed1ab_0
```

See PR Ouranosinc/PAVICS-e2e-workflow-tests#75
for more details.
@tlvu tlvu merged commit fae2468 into master Jun 15, 2021
@tlvu tlvu deleted the new-docker-build-for-ravenpy branch June 15, 2021 21:34
tlvu added a commit to bird-house/finch that referenced this pull request Jun 15, 2021
@tlvu
Merge pull request #188 from bird-house/update-finch-usage.ipynb-outp…
756ce6a 
…ut-for-jenkins

finch-usage.ipynb: update output due to new birdy 0.8.0

New birdy removed `output_formats`.

Clarification from DavidH: To clarify, the `output_formats` option still exists, but it's only appearing when the process has a complexoutput supporting more than one data format. 
bird-house/birdy@2bea46c#diff-ba0f245ac7a3747697d42961f7a19568e294dc5d2b4ad14b62a47d8c240ed12eR230-R238

Also removed `NBVAL_IGNORE_OUTPUT` since Jenkins is configured to by
default ignore html changes, see PR Ouranosinc/PAVICS-e2e-workflow-tests#73.

Jenkins passing build with upcoming Jupyter env: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/10/console

New Jupyter env PR: Ouranosinc/PAVICS-e2e-workflow-tests#75

To fix this error:
```
  _________ finch-master/docs/source/notebooks/finch-usage.ipynb::Cell 1 _________
  Notebook cell execution failed
  Cell 1: Cell outputs differ

  Input:
  help(wps.frost_days)

  Traceback:
   mismatch 'stdout'

   assert reference_output == test_output failed:

    'Help on meth...ut files.\n\n' == 'Help on meth...ut files.\n\n'
    Skipping 201 identical leading characters in diff, use -v to show
    - iable=None) method of birdy.client.base.WPSClient instance
    + iable=None, output_formats=None) method of birdy.client.base.WPSClient instance
    ?           +++++++++++++++++++++
          Number of days where daily minimum temperatures are below 0.

          Parameters
          ----------
          tasmin : ComplexData:mimetype:`application/x-netcdf`, :mimetype:`application/x-ogc-dods`
              NetCDF Files or archive (tar/zip) containing netCDF files.
          thresh : string
              Freezing temperature.
          freq : {'YS', 'MS', 'QS-DEC', 'AS-JUL'}string
              Resampling frequency.
          check_missing : {'any', 'wmo', 'pct', 'at_least_n', 'skip', 'from_context'}string
              Method used to determine which aggregations should be considered missing.
          missing_options : ComplexData:mimetype:`application/json`
              JSON representation of dictionary of missing method parameters.
          cf_compliance : {'log', 'warn', 'raise'}string
              Whether to log, warn or raise when inputs have non-CF-compliant attributes.
          data_validation : {'log', 'warn', 'raise'}string
              Whether to log, warn or raise when inputs fail data validation checks.
          variable : string
              Name of the variable in the NetCDF file.

          Returns
          -------
          output_netcdf : ComplexData:mimetype:`application/x-netcdf`
              The indicator values computed on the original input grid.
          output_log : ComplexData:mimetype:`text/plain`
              Collected logs during process run.
          ref : ComplexData:mimetype:`application/metalink+xml; version=4.0`
              Metalink file storing all references to output files.
```
tlvu added a commit to Ouranosinc/pavics-sdi that referenced this pull request Jun 15, 2021
@tlvu
Merge pull request #222 from Ouranosinc/update-subsetting.ipynb-outpu…
651ec84 
…t-for-jenkins

subsetting.ipynb: update output due to new birdy 0.8.0

New birdy removed `output_formats`.

Clarification from DavidH: To clarify, the `output_formats` option still exists, but it's only appearing when the process has a complexoutput supporting more than one data format. 
bird-house/birdy@2bea46c#diff-ba0f245ac7a3747697d42961f7a19568e294dc5d2b4ad14b62a47d8c240ed12eR230-R238

Also removed `NBVAL_IGNORE_OUTPUT` since Jenkins is configured to by
default ignore html changes, see PR
Ouranosinc/PAVICS-e2e-workflow-tests#73.

Jenkins passing build with upcoming Jupyter env: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-for-ravenpy/10/console

New Jupyter env PR: Ouranosinc/PAVICS-e2e-workflow-tests#75

To fix this error:
```
  _______ pavics-sdi-master/docs/source/notebooks/subsetting.ipynb::Cell 2 _______
  Notebook cell execution failed
  Cell 2: Cell outputs differ

  Input:
  help(fp.subset_continents)

  Traceback:
   mismatch 'stdout'

   assert reference_output == test_output failed:

    'Help on meth... outputs.\n\n' == 'Help on meth... outputs.\n\n'
    Skipping 101 identical leading characters in diff, use -v to show
    - n='Africa') method of birdy.client.base.WPSClient instance
    + n='Africa', output_formats=None) method of birdy.client.base.WPSClient instance
    ?           +++++++++++++++++++++
          Return the data whose grid cells intersect the selected continents for each input dataset.

          Parameters
          ----------
          region : {'Africa', 'Asia', 'Australia', 'North America', 'Oceania', 'South America', 'Antarctica', 'Europe'}string
              Continent name.
          resource : ComplexData:mimetype:`application/x-netcdf`, :mimetype:`application/x-tar`, :mimetype:`application/zip`
              NetCDF Files or archive (tar/zip) containing netCDF files.

          Returns
          -------
          output : ComplexData:mimetype:`application/x-netcdf`
              NetCDF output for first resource file.
          metalink : ComplexData:mimetype:`application/metalink+xml; version=4.0`
              Metalink file with links to all NetCDF outputs.
```
tlvu added a commit to bird-house/birdhouse-deploy that referenced this pull request Jun 15, 2021
@tlvu
Merge pull request #175 from bird-house/new-jupyter-env-for-ravenpy-a…
1586e98 
…nd-birdy

jupyter: new version for updated ravenpy, birdy and xclim

PR to deploy the new Jupyter env to PAVICS.

See PR Ouranosinc/PAVICS-e2e-workflow-tests#75
for more details.

Relevant changes:

```diff
<   - ravenpy=0.4.2=py37_1
>   - ravenpy=0.5.2=pyh7f9bfb9_0

# Renamed.
<   - raven=3.0.4.318=hc9bffa2_2
>   - raven-hydro=3.0.4.322=h516393e_0

<   - ostrich=21.03.16=h2bc3f7f_0
>   - ostrich=21.03.16=h4bd325d_1

<   - xclim=0.25.0=pyhd8ed1ab_0
>   - xclim=0.27.0=pyhd8ed1ab_0

# Old version was from pip.
<     - birdhouse-birdy==0.7.0
>   - birdy=v0.8.0=pyh6c4a22f_0

# Was previously included in another package, now it is standalone.
>   - pydantic=1.8.2=py37h5e8e339_0

# New libs for upcoming Raven notebooks
>   - gcsfs=2021.6.0=pyhd8ed1ab_0
>   - intake=0.6.2=pyhd8ed1ab_0
>   - intake-esm=2021.1.15=pyhd8ed1ab_0
>   - zarr=2.8.3=pyhd8ed1ab_0

<   - xarray=0.17.0=pyhd8ed1ab_0
>   - xarray=0.18.2=pyhd8ed1ab_0

<   - owslib=0.23.0=pyhd8ed1ab_0
>   - owslib=0.24.1=pyhd8ed1ab_0

<   - cf_xarray=0.5.1=pyh44b312d_0
>   - cf_xarray=0.5.2=pyh6c4a22f_0

<   - clisops=0.6.3=pyh44b312d_0
>   - clisops=0.6.5=pyh6c4a22f_0

<   - dask=2021.2.0=pyhd8ed1ab_0
>   - dask=2021.6.0=pyhd8ed1ab_0

# Downgrade !
<   - gdal=3.2.1=py37hc5bc4e4_7
>   - gdal=3.1.4=py37h2ec2946_8

# Downgrade !
<   - rasterio=1.2.2=py37hd5c4cce_0
>   - rasterio=1.2.1=py37ha549118_0

<   - hvplot=0.7.1=pyh44b312d_0
>   - hvplot=0.7.2=pyh6c4a22f_0

<   - rioxarray=0.3.1=pyhd8ed1ab_0
>   - rioxarray=0.4.1.post0=pyhd8ed1ab_0

# Downgrade !
<   - xskillscore=0.0.19=pyhd8ed1ab_0                                                                                                                 
>   - xskillscore=0.0.18=py_1
```

Full diff of `conda env export`:
[210415-210527.1-update210615-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6658638/210415-210527.1-update210615-conda-env-export.diff.txt)

Full new `conda env export`:
[210527.1-update210615-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6658646/210527.1-update210615-conda-env-export.yml.txt)
tlvu added a commit to Ouranosinc/raven that referenced this pull request Jun 15, 2021
@tlvu
Merge pull request #393 from Ouranosinc/fix-nb-for-new-jupyter-env
e0af129 
Extract_geographical_watershed_properties.ipynb: fix run failure in new Jupyter env

Fixes #392 using hints from #388. Closes #388.

Note I am not sure what I am doing as I do not understand the code.  I simply try to get around the errors and produce similar graph.  The old graph had `spatial_ref` in the title, the new one don't.  Wonder if it's something critical.

Clarification from Trevor: 

 ==========
Putting it simply, the object returned (`xarray.DataArray`) is different from what the following cell needed to handle for (`rasterio._io.RasterReader` → `numpy.ndarray`):
```
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
```
The `spatial_ref`title is something that `rasterio` adds to the plot, and it isn't at all needed. The notebook should work perfectly so long as the client running it has `rioxarray`available in their ipython kernel.

 ==========

With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below.

```
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 7: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  ValueError                                Traceback (most recent call last)
23:00:58  <ipython-input-8-8b167a3ecdc9> in <module>
23:00:58        4 features, statistics, grid = stats_resp.get(asobj=True)
23:00:58        5
23:00:58  ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58        7     with memfile.open() as dataset:
23:00:58        8         grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext)
23:00:58      103         """
23:00:58      104         super(MemoryFile, self).__init__(
23:00:58  --> 105             file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext)
23:00:58      106
23:00:58      107     @ensure_env
23:00:58
23:00:58  rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__()
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self)
23:00:58      127
23:00:58      128     def __bool__(self: Any) -> bool:
23:00:58  --> 129         return bool(self.values)
23:00:58      130
23:00:58      131     def __float__(self: Any) -> float:
23:00:58
23:00:58  ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 10: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58  display(unique, counts)
23:00:58
23:00:58  # Pixels values at '127' are NaN and can be ignored.
23:00:58  plt.bar(unique[:-1], counts[:-1])
23:00:58  plt.show()
23:00:58
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-11-a26c52126e9c> in <module>
23:00:58  ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58        2 display(unique, counts)
23:00:58        3
23:00:58        4 # Pixels values at '127' are NaN and can be ignored.
23:00:58        5 plt.bar(unique[:-1], counts[:-1])
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'sel'
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 11: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  import cartopy.crs as ccrs
23:00:58
23:00:58  # Set a CRS transformation:
23:00:58  crs = ccrs.LambertConformal(
23:00:58      central_latitude=49, central_longitude=-95, standard_parallels=(49, 77)
23:00:58  )
23:00:58
23:00:58  ax = plt.subplot(projection=crs)
23:00:58  grid.name = "Land Use Categories"
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-12-f9fdb6f0e562> in <module>
23:00:58        7
23:00:58        8 ax = plt.subplot(projection=crs)
23:00:58  ----> 9 grid.name = "Land Use Categories"
23:00:58       10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58       11 plt.show()
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'name'
```
@tlvu tlvu mentioned this pull request Jun 18, 2021
Merged
tlvu added a commit that referenced this pull request Jun 18, 2021
@tlvu
Merge pull request #76 from Ouranosinc/new-docker-build-raven-notebooks
3806e67 
New docker build for raven notebooks

Add `intake-xarray` (fixes Ouranosinc/pavics-sdi#223)

@huard @tlogan2000 also proactively add `intake-geopandas` and `intake-thredds` as they look interesting.  Tried to add `intake-stac` but got into conflicting dependencies during build so had to leave it out.

This is an incremental build (not a full build) from the previous release (#75) so only those are changed (so regression risk is extremely low):
```
    package                    |            build                                                                                                     
    ---------------------------|-----------------                                                                                                     
    intake-geopandas-0.2.4     |     pyhd8ed1ab_0          12 KB  conda-forge                                                                         
    intake-thredds-2021.6.16   |     pyhd8ed1ab_0          13 KB  conda-forge                                                                         
    intake-xarray-0.5.0        |     pyhd8ed1ab_0         1.4 MB  conda-forge                                                                         
    ------------------------------------------------------------
```

Jenkins build with only known error: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-raven-notebooks/1/console

Raven notebooks passing: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build-raven-notebooks/2/console

Manual tested Pavics homepage notebooks as well, using https://medus.ouranos.ca/jupyter since this image is deployed there.

Matching PR to deploy to PAVICS bird-house/birdhouse-deploy#178
Zeitsperre pushed a commit to Ouranosinc/raven that referenced this pull request Aug 17, 2023
@tlvu
Extract_geographical_watershed_properties.ipynb: fix run failure in n…
91a0790 
…ew Jupyter env

Fixes #392.

With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below.

```
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 7: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  ValueError                                Traceback (most recent call last)
23:00:58  <ipython-input-8-8b167a3ecdc9> in <module>
23:00:58        4 features, statistics, grid = stats_resp.get(asobj=True)
23:00:58        5
23:00:58  ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58        7     with memfile.open() as dataset:
23:00:58        8         grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext)
23:00:58      103         """
23:00:58      104         super(MemoryFile, self).__init__(
23:00:58  --> 105             file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext)
23:00:58      106
23:00:58      107     @ensure_env
23:00:58
23:00:58  rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__()
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self)
23:00:58      127
23:00:58      128     def __bool__(self: Any) -> bool:
23:00:58  --> 129         return bool(self.values)
23:00:58      130
23:00:58      131     def __float__(self: Any) -> float:
23:00:58
23:00:58  ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 10: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58  display(unique, counts)
23:00:58
23:00:58  # Pixels values at '127' are NaN and can be ignored.
23:00:58  plt.bar(unique[:-1], counts[:-1])
23:00:58  plt.show()
23:00:58
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-11-a26c52126e9c> in <module>
23:00:58  ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58        2 display(unique, counts)
23:00:58        3
23:00:58        4 # Pixels values at '127' are NaN and can be ignored.
23:00:58        5 plt.bar(unique[:-1], counts[:-1])
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'sel'
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 11: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  import cartopy.crs as ccrs
23:00:58
23:00:58  # Set a CRS transformation:
23:00:58  crs = ccrs.LambertConformal(
23:00:58      central_latitude=49, central_longitude=-95, standard_parallels=(49, 77)
23:00:58  )
23:00:58
23:00:58  ax = plt.subplot(projection=crs)
23:00:58  grid.name = "Land Use Categories"
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-12-f9fdb6f0e562> in <module>
23:00:58        7
23:00:58        8 ax = plt.subplot(projection=crs)
23:00:58  ----> 9 grid.name = "Land Use Categories"
23:00:58       10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58       11 plt.show()
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'name'
```
Zeitsperre pushed a commit to Ouranosinc/raven that referenced this pull request Aug 17, 2023
@tlvu
Merge pull request #393 from Ouranosinc/fix-nb-for-new-jupyter-env
8785a9e 
Extract_geographical_watershed_properties.ipynb: fix run failure in new Jupyter env

Fixes #392 using hints from #388. Closes #388.

Note I am not sure what I am doing as I do not understand the code.  I simply try to get around the errors and produce similar graph.  The old graph had `spatial_ref` in the title, the new one don't.  Wonder if it's something critical.

Clarification from Trevor: 

 ==========
Putting it simply, the object returned (`xarray.DataArray`) is different from what the following cell needed to handle for (`rasterio._io.RasterReader` → `numpy.ndarray`):
```
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
```
The `spatial_ref`title is something that `rasterio` adds to the plot, and it isn't at all needed. The notebook should work perfectly so long as the client running it has `rioxarray`available in their ipython kernel.

 ==========

With the new Jupypter env for latest RavenPy `0.5.2` from PR Ouranosinc/PAVICS-e2e-workflow-tests#75 `Extract_geographical_watershed_properties.ipynb` fails with the following error below.

```
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 7 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 7: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  # Note that geojson, and rasterio/rioxarray need to be installed for this to work.
23:00:58  # In the next birdy release, the tiff grid will be automatically converted to a DataArray.
23:00:58  # Here we need to manually convert it.
23:00:58  features, statistics, grid = stats_resp.get(asobj=True)
23:00:58
23:00:58  with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58      with memfile.open() as dataset:
23:00:58          grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  grid.plot()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  ValueError                                Traceback (most recent call last)
23:00:58  <ipython-input-8-8b167a3ecdc9> in <module>
23:00:58        4 features, statistics, grid = stats_resp.get(asobj=True)
23:00:58        5
23:00:58  ----> 6 with rasterio.io.MemoryFile(grid[0]) as memfile:
23:00:58        7     with memfile.open() as dataset:
23:00:58        8         grid = rio.open_rasterio(dataset)
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/rasterio/io.py in __init__(self, file_or_bytes, dirname, filename, ext)
23:00:58      103         """
23:00:58      104         super(MemoryFile, self).__init__(
23:00:58  --> 105             file_or_bytes=file_or_bytes, dirname=dirname, filename=filename, ext=ext)
23:00:58      106
23:00:58      107     @ensure_env
23:00:58
23:00:58  rasterio/_io.pyx in rasterio._io.MemoryFileBase.__init__()
23:00:58
23:00:58  /opt/conda/envs/birdy/lib/python3.7/site-packages/xarray/core/common.py in __bool__(self)
23:00:58      127
23:00:58      128     def __bool__(self: Any) -> bool:
23:00:58  --> 129         return bool(self.values)
23:00:58      130
23:00:58      131     def __float__(self: Any) -> float:
23:00:58
23:00:58  ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 10 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 10: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58  display(unique, counts)
23:00:58
23:00:58  # Pixels values at '127' are NaN and can be ignored.
23:00:58  plt.bar(unique[:-1], counts[:-1])
23:00:58  plt.show()
23:00:58
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-11-a26c52126e9c> in <module>
23:00:58  ----> 1 unique, counts = numpy.unique(grid.sel(band=1), return_counts=True)
23:00:58        2 display(unique, counts)
23:00:58        3
23:00:58        4 # Pixels values at '127' are NaN and can be ignored.
23:00:58        5 plt.bar(unique[:-1], counts[:-1])
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'sel'
23:00:58
23:00:58  _ raven-master/docs/source/notebooks/Extract_geographical_watershed_properties.ipynb::Cell 11 _
23:00:58  Notebook cell execution failed
23:00:58  Cell 11: Cell execution caused an exception
23:00:58
23:00:58  Input:
23:00:58  import cartopy.crs as ccrs
23:00:58
23:00:58  # Set a CRS transformation:
23:00:58  crs = ccrs.LambertConformal(
23:00:58      central_latitude=49, central_longitude=-95, standard_parallels=(49, 77)
23:00:58  )
23:00:58
23:00:58  ax = plt.subplot(projection=crs)
23:00:58  grid.name = "Land Use Categories"
23:00:58  grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58  plt.show()
23:00:58
23:00:58  Traceback:
23:00:58
23:00:58  ---------------------------------------------------------------------------
23:00:58  AttributeError                            Traceback (most recent call last)
23:00:58  <ipython-input-12-f9fdb6f0e562> in <module>
23:00:58        7
23:00:58        8 ax = plt.subplot(projection=crs)
23:00:58  ----> 9 grid.name = "Land Use Categories"
23:00:58       10 grid.where(grid != 127).sel(band=1).plot.imshow(ax=ax, transform=crs, cmap="tab20")
23:00:58       11 plt.show()
23:00:58
23:00:58  AttributeError: 'list' object has no attribute 'name'
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@huard huard huard approved these changes

@cjauvin cjauvin cjauvin approved these changes

@aulemahal aulemahal Awaiting requested review from aulemahal

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Need to install new packages on the PAVICS server for notebook features
5 participants
@tlvu @aulemahal @tlogan2000 @huard @cjauvin