docker: update to JupyterLab v3 and fix broken jupyterlab-system-monitor plugin #85
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…m-monitor jupyterlab-system-monitor has not been working with jupyterlab v2
…t jupyterlab v3
```
Step 11/20 : RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager && jupyter serverextension enable voila --sys-prefix && jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leaflet && jupyter labextension install @jupyterlab/debugger && jupyter labextension install @jupyterlab/google-drive && jupyter labextension install jupyterlab-topbar-text jupyterlab_conda
---> Running in 96e4392006b2
Building jupyterlab assets (production, minimized)
Enabling: voila
- Writing config: /opt/conda/envs/birdy/etc/jupyter
- Validating...
voila 0.2.10 OK
Building jupyterlab assets (production, minimized)
Building jupyterlab assets (production, minimized)
\ An error occured.
ValueError: The extension "@jupyterlab/google-drive" does not yet support the current version of JupyterLab.
Conflicting Dependencies:
JupyterLab Extension Package
>=3.0.11 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/application
>=3.0.9 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/apputils
>=5.0.6 <5.1.0 >=4.0.0 <5.0.0 @jupyterlab/coreutils
>=3.0.11 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/docmanager
>=3.0.11 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/filebrowser
>=3.0.9 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/mainmenu
>=6.0.9 <6.1.0 >=5.0.0 <6.0.0 @jupyterlab/services
>=3.0.6 <3.1.0 >=2.0.0 <3.0.0 @jupyterlab/settingregistry
See the log file for details: /tmp/jupyterlab-debug-shgkoz6v.log
The command '/bin/sh -c jupyter labextension install @jupyter-widgets/jupyterlab-manager && jupyter serverextension enable voila --sys-prefix && jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leaflet && jupyter labextension install @jupyterlab/debugger && jupyter labextension install @jupyterlab/google-drive && jupyter labextension install jupyterlab-topbar-text jupyterlab_conda' returned a non-zero code: 1
```
…lab_pyviz in jupyterlab v2
tlvu
added a commit
to Ouranosinc/pavics-sdi
that referenced
this pull request
Jul 28, 2021
New Jupyter env in Ouranosinc/PAVICS-e2e-workflow-tests#85. This is still weird as we are already ignoring by default all html changes in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/3172f621afe991c69caa0fa10c7d40f2f09fdd02/conftest.py#L5 Error fixed: ``` _______ pavics-sdi-master/docs/source/notebooks/subsetting.ipynb::Cell 4 _______ Notebook cell execution failed Cell 4: Cell outputs differ Input: res = resp.get() print("URL: ", res.output) res = resp.get(asobj=True) res.output Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data... 2.6.0' == '<xarray.Data... 2.6.0' Skipping 1136 identical trailing characters in diff, use -v to show <xarray.Dataset> - Dimensions: (time: 12, bnds: 2, lat: 38, lon: 37) + Dimensions: (bnds: 2, lat: 38, lon: 37, time: 12) Coordin ```
tlvu
added a commit
to bird-house/finch
that referenced
this pull request
Jul 28, 2021
New Jupyter env in Ouranosinc/PAVICS-e2e-workflow-tests#85 This is a bit weird since we already ignore all html output changes in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/3172f621afe991c69caa0fa10c7d40f2f09fdd02/conftest.py#L5 Error fixed: ``` _________ finch-master/docs/source/notebooks/finch-usage.ipynb::Cell 5 _________ Notebook cell execution failed Cell 5: Cell outputs differ Input: out = resp.get(asobj=True) out.output_netcdf Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data... CCCS' == '<xarray.Data... CCCS' Skipping 1109 identical trailing characters in diff, use -v to show <xarray.Dataset> - Dimensions: (time: 20, lat: 6, lon: 7) + Dimensions: (lat: 6, lon: 7, time: 20) Coordin ```
tlvu
added a commit
to Ouranosinc/PAVICS-landing
that referenced
this pull request
Jul 28, 2021
New Jupyter env in Ouranosinc/PAVICS-e2e-workflow-tests#85. Funny that I have to add NBVAL_IGNORE_OUTPUT to a cell that has empty output! Error fixed: ``` _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb::Cell 3 _ Notebook cell execution failed Cell 3: Cell outputs differ Input: # Start with 30 year means from mapping step # Calculate weighted average for each region as in last step with xr.set_options(keep_attrs=True): reg30y_sims = (d30yAvg * weight_masks).sum(dim=['lat','lon']) reg30y_ens = xens.ensemble_percentiles(reg30y_sims) # make the 'horizon' field the index of our table. rename the 'geom' field to 'region' df = reg30y_ens.swap_dims(dict(year='horizon')).rename({'geom':'region'}).to_dataframe() df.hvplot.table(title='Summary table',fit_columns=False, sortable=True, selectable=True, width=2000) Traceback: mismatch 'text/plain' assert reference_output == test_output failed: ':Table [ho...ve_delta_p90]' == ':Table [rc...ve_delta_p90]' Skipping 227 identical trailing characters in diff, use -v to show - :Table [rcp,horizon,season,region,tx_mea + :Table [horizon,rcp,region,season,tx_mea ```
This was referenced Jul 28, 2021
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Jul 28, 2021
…enPy-0.7.0 See PR Ouranosinc/PAVICS-e2e-workflow-tests#85 for more info. Relevant changes: ```diff < - jupyterlab=2.2.9=pyhd8ed1ab_0 > - jupyterlab=3.1.0=pyhd8ed1ab_0 < - jupyterlab_server=1.2.0=py_0 > - jupyterlab_server=2.6.1=pyhd8ed1ab_0 < - jupyter-archive=2.2.0=pyhd8ed1ab_0 > - jupyter-archive=3.0.1=pyhd8ed1ab_0 < - jupyter_bokeh=2.0.4=pyhd8ed1ab_0 > - jupyter_bokeh=3.0.2=pyhd8ed1ab_0 < - jupyterlab-git=0.24.0=pyhd8ed1ab_0 > - jupyterlab-git=0.31.0=pyhd8ed1ab_0 < - nbdime=2.1.0=py_0 > - nbdime=3.1.0=pyhd8ed1ab_0 < - nbresuse==0.4.0 > - nbresuse=0.4.0=pyhd8ed1ab_0 > - nbclassic=0.3.1=pyhd8ed1ab_1 > - jupyterlab-system-monitor=0.8.0=pyhd8ed1ab_1 > - jupyter-resource-usage=0.5.1=pyhd8ed1ab_0 > - jupyterlab-topbar=0.6.1=pyhd8ed1ab_2 > - jupyterlab-logout=0.5.0 < - jupyter_conda=5.1.1=hd8ed1ab_0 < - ravenpy=0.6.0=pyh1bb2064_2 > - ravenpy=0.7.0=pyh1bb2064_0 < - pandas=1.2.5=py37h219a48f_0 > - pandas=1.3.1=py37h219a48f_0 < - xarray=0.18.2=pyhd8ed1ab_0 > - xarray=0.19.0=pyhd8ed1ab_1 < - dask=2021.7.0=pyhd8ed1ab_0 > - dask=2021.7.1=pyhd8ed1ab_0 < - regionmask=0.6.2=pyhd8ed1ab_0 > - regionmask=0.7.0=pyhd8ed1ab_0 ```
|
All Jenkins passed, this PR is ready for review. |
matprov
approved these changes
Jul 28, 2021
tlvu
added a commit
to Ouranosinc/pavics-sdi
that referenced
this pull request
Jul 29, 2021
subsetting.ipynb: update output for new Jupyter env New Jupyter env in Ouranosinc/PAVICS-e2e-workflow-tests#85. This is still weird as we are already ignoring by default all html changes in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/3172f621afe991c69caa0fa10c7d40f2f09fdd02/conftest.py#L5 Error fixed: ``` _______ pavics-sdi-master/docs/source/notebooks/subsetting.ipynb::Cell 4 _______ Notebook cell execution failed Cell 4: Cell outputs differ Input: res = resp.get() print("URL: ", res.output) res = resp.get(asobj=True) res.output Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data... 2.6.0' == '<xarray.Data... 2.6.0' Skipping 1136 identical trailing characters in diff, use -v to show <xarray.Dataset> - Dimensions: (time: 12, bnds: 2, lat: 38, lon: 37) + Dimensions: (bnds: 2, lat: 38, lon: 37, time: 12) Coordin ```
tlvu
added a commit
to bird-house/finch
that referenced
this pull request
Jul 29, 2021
finch-usage.ipynb: update output for new jupyter env New Jupyter env in Ouranosinc/PAVICS-e2e-workflow-tests#85 This is a bit weird since we already ignore all html output changes in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/3172f621afe991c69caa0fa10c7d40f2f09fdd02/conftest.py#L5 Error fixed: ``` _________ finch-master/docs/source/notebooks/finch-usage.ipynb::Cell 5 _________ Notebook cell execution failed Cell 5: Cell outputs differ Input: out = resp.get(asobj=True) out.output_netcdf Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data... CCCS' == '<xarray.Data... CCCS' Skipping 1109 identical trailing characters in diff, use -v to show <xarray.Dataset> - Dimensions: (time: 20, lat: 6, lon: 7) + Dimensions: (lat: 6, lon: 7, time: 20) Coordin ```
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Jul 29, 2021
…b-v3 jupyter: update for JupyterLab v3, fix memory monitor display and RavenPy-0.7.0 See PR Ouranosinc/PAVICS-e2e-workflow-tests#85 for more info. Relevant changes: ```diff < - jupyterlab=2.2.9=pyhd8ed1ab_0 > - jupyterlab=3.1.0=pyhd8ed1ab_0 < - jupyterlab_server=1.2.0=py_0 > - jupyterlab_server=2.6.1=pyhd8ed1ab_0 < - jupyter-archive=2.2.0=pyhd8ed1ab_0 > - jupyter-archive=3.0.1=pyhd8ed1ab_0 < - jupyter_bokeh=2.0.4=pyhd8ed1ab_0 > - jupyter_bokeh=3.0.2=pyhd8ed1ab_0 < - jupyterlab-git=0.24.0=pyhd8ed1ab_0 > - jupyterlab-git=0.31.0=pyhd8ed1ab_0 < - nbdime=2.1.0=py_0 > - nbdime=3.1.0=pyhd8ed1ab_0 < - nbresuse==0.4.0 > - nbresuse=0.4.0=pyhd8ed1ab_0 > - nbclassic=0.3.1=pyhd8ed1ab_1 > - jupyterlab-system-monitor=0.8.0=pyhd8ed1ab_1 > - jupyter-resource-usage=0.5.1=pyhd8ed1ab_0 > - jupyterlab-topbar=0.6.1=pyhd8ed1ab_2 > - jupyterlab-logout=0.5.0 < - jupyter_conda=5.1.1=hd8ed1ab_0 < - ravenpy=0.6.0=pyh1bb2064_2 > - ravenpy=0.7.0=pyh1bb2064_0 < - pandas=1.2.5=py37h219a48f_0 > - pandas=1.3.1=py37h219a48f_0 < - xarray=0.18.2=pyhd8ed1ab_0 > - xarray=0.19.0=pyhd8ed1ab_1 < - dask=2021.7.0=pyhd8ed1ab_0 > - dask=2021.7.1=pyhd8ed1ab_0 < - regionmask=0.6.2=pyhd8ed1ab_0 > - regionmask=0.7.0=pyhd8ed1ab_0 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
JupyterLab v3 was available since January 2021 but we had to pin to v2 since many of the plugins we use were broken at the time for v3.
Fast forward 7 months later, only the Google Drive plugin still do not support JupyterLab v3 so we decided to drop it since it was only used by one single user (
ls -larth /data/jupyterhub_user_data/*/.home/.jupyter/lab/user-settings/@jupyterlab/google-drive /dev/nullshows who uses it). We have not enabled the system-wide Google Drive config due to security concern so each user had to configure it themselves separately.Staying on JupyterLab v2 caused us the reverse problem, some plugins stopped working with v2, example the jupyterlab-system-monitor one. That one is becoming more crucial as we have more and more users and we would like users to be aware of their resource consumptions on the system.
This PR fixes #83, the jupyterlab-system-monitor problem.
Changes
Additional Information
Screenshot showing differences between v2 vs v3 and the working jupyterlab-system-monitor plugin.
Bokeh and pyviz testing by running
ClimateDataAnalysis-5Visualization.ipynbon the old and new Jupyter env and manually comparing the graphics generated. They look similar.Jenkins build all passed https://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/update-to-jupyterlab-v3/7/console (CRIM Jenkins)
Notebooks fixes required for the above passing Jenkins build:
Jenkins build for Raven nb, all passed: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/update-to-jupyterlab-v3/5/console
Matching PR to deploy this new Jupyter env to PAVICS: bird-house/birdhouse-deploy#185
Full diff of
conda env export:210723-210728-conda-env-export.diff.txt
Full new
conda env export:210728-conda-env-export.yml.txt