TFMA not rendering in JupyterLab #112

ConverJens · 2021-02-19T13:56:57Z

System information

Have I written custom code (as opposed to using a stock example script
provided in TensorFlow Model Analysis): Yes, minor
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04 and image python:3.7-slim
TensorFlow Model Analysis installed from (source or binary): pip install
TensorFlow Model Analysis version (use command below): 0.27.0
Python version: 3.7
Jupyter Notebook version: jupyterlab 2.2.9
Exact command to reproduce: see sample notebook

Describe the problem

tfma.view.render_slicing_metrics shows no output.

Source code / logs

Slim docker image to reproduce the issue:

FROM python:3.7-slim

ENV DEBIAN_FRONTEND=noninteractive

# This is used because our k8s cluster can only access our internal pypi
#COPY pip.conf /etc/pip.conf

# # TFMA is installed in the notebook because pip complained otherwise
RUN python3.7 -m pip install --no-cache-dir jupyterlab==2.2.9

# Install Node (for jupyter lab extensions)
RUN apt update && \
    apt -y install nano curl dirmngr apt-transport-https lsb-release ca-certificates && \
    curl -L https://deb.nodesource.com/setup_15.x | bash - && \
    apt update && apt install -y nodejs && \
    node -v

RUN jupyter labextension install tensorflow_model_analysis@0.27.0 && \
    jupyter labextension install @jupyter-widgets/jupyterlab-manager@2

RUN jupyter lab build

ENV DEBIAN_FRONTEND=

ENV NB_PREFIX /
ENV SHELL=/bin/bash

# Standard KubeFlow jupyter startup command
CMD ["/bin/bash","-c", "jupyter lab --notebook-dir=/home/jovyan --ip=0.0.0.0 --no-browser --allow-root --port=8888 --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='*' --NotebookApp.base_url=${NB_PREFIX}"]

Below is an evaluation artifact from a small TFX pipeline and a minimal notebook to reproduce. Notebook consists of unzipping, install tfma, load eval result and try to display.
3053.zip
tfma-render-issue.ipynb.zip

Edit: I've also run the above with node 12 instead of 15 with the exact same result.

The text was updated successfully, but these errors were encountered:

fhuanming · 2021-02-19T19:21:57Z

cc: @atn832

atn832 · 2021-02-22T08:06:32Z

Thanks for sharing the steps. I haven't tried reproducing yet, but judging from the error you posted at #56 (comment), it seems like the TFMA extension is downloading vulcanized_tfma.js from the wrong URL. As a result, instead of actual Javascript, it gets some kind of markup, probably something like:

<!doctype html>
<html>
  ...
  file not found
  ...
</html>

It reads the first < and returns this error:

Uncaught SyntaxError: Unexpected token '<' vulcanized_tfma.js:1

In the Network tab of the Chrome debugger tool, can you check the url of the request for vulcanized_tfma.js and share it here?

ConverJens · 2021-02-22T08:31:53Z

@atn832 If I understand correctly, the url is: :31380/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js

Is TFMA trying to download anything? We are running in an on-prem cluster with no external access so if TFMA needs external access that would be an issue.

Bobgy · 2021-02-25T11:22:20Z

Getting the same issue in kubeflow/pipelines#5194, we use technique in #10 (comment) to visualize TFMA as html and then embed that HTML in a different place in iframe.

The problem is similar to this reported issue that vulcanized_tfma.js doesn't load. I verified that TFMA 0.26.0 still works for us, so the regression happens between the two versions.

ConverJens · 2021-02-25T11:34:50Z

@Bobgy Is the trick with embedding the html required? This is not something I usually use, for instance when rendering stats from TFDV.

Bobgy · 2021-02-26T09:55:53Z

My use case

I generate the html in a step in a pipeline, and the pipeline UI shows the html in an iframe. but that's not related to this issue

fhuanming · 2021-03-01T18:04:11Z

Hi @ConverJens, besides what @atn832 asked, I have run the dockerfile with the notebook and data that you shared above (by docker run -p 8888:8888 {image_name}). I can successfully load the TFMA UI on the JupyterLab.

In terms of

Is TFMA trying to download anything?

I don't know what the answer is. Maybe @atn832 can give the answer. I only know that running jupyter labextension install tensorflow_model_analysis@0.27.0 will download TFMA js packages from NPM.

atn832 · 2021-03-02T04:52:59Z

Upon running render_slicing_metrics in a cell, Chrome will download vulcanized_tfma.js. This is hosted by Jupyter Lab once you install the TFMA extension, so it should work even without internet.

@ConverJens, can you share what you see in the Response tab for the vulcanized_tfma.js request? I expect it'll show some HTML with an error message that might tell us what is wrong.

On my machine, since it worked, I can see a bunch of Javascript comments followed by Javascript code like this:

ConverJens · 2021-03-02T09:28:02Z

@atn832 You are completely right! The response I get is basically Please enable JavaScript to view this website. which is very weird since my chrome settings says it's allowed for all sites, which is the recommended setting. I'm using latest version of Chrome (88). Full response below. I get the same response when using Safari.

Any idea how to proceed?

@Bobgy You mentioned that you had this issue in KubeFlow as well. Is there some interaction with in KF that leads the browser into believing that JavaScript is disabled or otherwise may be causing this issue?

<!doctype html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1,user-scalable=yes"><meta name="description" content="Kubeflow Central Dashboard"><meta name="theme-color" content="#3f51b5"><title>Kubeflow Central Dashboard</title><link rel="shortcut icon" href="/assets/favicon.ico"><link rel="icon" href="/assets/favicon-32x32.png" sizes="32x32"><link rel="icon" href="/assets/favicon-57x57.png" sizes="57x57"><link rel="icon" href="/assets/favicon-76x76.png" sizes="76x76"><link rel="icon" href="/assets/favicon-96x96.png" sizes="96x96"><link rel="icon" href="/assets/favicon-128x128.png" sizes="128x128"><link rel="icon" href="/assets/favicon-192x192.png" sizes="192x192"><link rel="apple-touch-icon" href="/assets/favicon-152x152.png" sizes="152x152"><link rel="apple-touch-icon" href="/assets/favicon-180x180.png" sizes="180x180"><base href="/"><script src="webcomponentsjs/webcomponents-loader.js"></script><script src="webcomponentsjs/custom-elements-es5-adapter.js"></script><link href="app.css" rel="stylesheet"></head><body><main-page></main-page><noscript>Please enable JavaScript to view this website.</noscript><script src="vendor.bundle.js" defer="defer"></script><script src="app.bundle.js" defer="defer"></script><script src="dashboard_lib.bundle.js" defer="defer"></script></body></html>

ConverJens · 2021-03-02T11:32:19Z

@fhuanming @Bobgy Note that I'm rendering TFMA works for me locally as well when running the image and data I supplied, but the exact same rendering fails in my KubeFlow hosted notebook. I'm don't know if this issue is because of something in KubeFlow or from the fact that our k8s cluster has no external access.

@atn832 Do you know if TFMA needs to be able to reach NPM at runtime?

@Bobgy Any ideas if KubeFlow itself is blocking something?

mwakaba2 · 2021-03-09T19:35:23Z

@atn832 Hi, I'm also running into a similar issue with a hosted notebook solution (not Kubeflow).
I'm also using the latest version of Chrome (88), and the notebook lives in a k8s cluster with no external access.
However, in my case, no response is returned.

When I try to render, this happens
Network request:

404 response:

I'd appreciate your help.

rcrowe-google · 2021-03-10T19:54:55Z

@mwakaba2 the fact that you're getting a 404 makes me wonder about where it's trying to load vulcanized_tfma.js from. Could you hover over the filename in the network request and give us the whole URL?

When I load that, it loads from:

https://localhost:8080/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js

which makes me wonder if this is a port problem, or a path problem, file permissions, or maybe an HTTPS problem (but I don't think that would be a 404). My guess is either a path or file permissions problem.

mwakaba2 · 2021-03-10T21:40:41Z

@rcrowe-google Yeah that's the path.

/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js

I think the issue is that our hosted notebook solution is using jupyter_server in the backend. Jupyter_server by default doesn't support the nbextensions path.
So to enable that, we have to enable nbclassic server extension.

I still get a 404 after that, so I think I also need to enable the extensions via nbextension install to add the "tensorflow_model_analysis/vulcanized_tfma.js" path.

mwakaba2 · 2021-03-10T22:59:49Z

Ok I got it working in Jupyterlab by installing and enabling the nbextensions.

 $ jupyter nbextension enable --py widgetsnbextension
 $ jupyter nbextension enable --py tensorflow_model_analysis

You need that in order to access the /nbextensions path.

rcrowe-google · 2021-03-11T02:27:15Z

Glad to hear you solved it Mariko! It seems so simple once you know the answer ... 😁

kwlzn · 2021-03-12T00:11:47Z

@rcrowe-google there's still an upstream problem here FYI.

in the Jupyter ecosystem, there are two primary types of frontend extensions (this doc may be helpful):

nbextensions - Jupyter Classic Notebook extensions - which support the classic Notebook UI.
labextensions - Jupyter Lab Extensions - which support the modern Jupyter Lab UI.

nbextensions are effectively deprecated in favor of labextensions, because all modern notebook runtimes (incl AI Platform Notebooks etc are fronted with Jupyter Lab). so in order to enable nbextensions (Jupyter Classic Notebook extensions) on top of Jupyter Server (which replaces the Jupyter Notebook server) to support serving of this Javascript, we had to install a compatibility layer that otherwise wouldn't need to exist (called nbclassic).

so tl;dr: the problem with TFMA's JupyterLab support is that it's not a pure JupyterLab extension. It's a labextension that depends on a nbextension to work. what we've employed here is a workaround vs a proper fix. the proper fix would involve making the labextension carry its own dependencies vs the existing hybrid model (which is unprecedented btw).

this gap is likely where the other folks like @ConverJens are running into issues making this work - as it's non-obvious vs the way any other lab extension works.

ConverJens · 2021-03-12T13:01:45Z

@mwakaba2 @rcrowe-google I tried the fix that you mentioned but there was no difference for me, still no output. Albeit, the error I had was different from the one @mwakaba2 experienced.

@kwlzn I tried installing nbclassic but with no progress. I used to start my server by running jupyter lab ... but should I start it with another command for this to have effect?

mwakaba2 · 2021-03-12T19:47:55Z

@ConverJens
You don't need nbclassic because jupyterlab 2.2.9 by default relies on the jupyter/notebook as the backend.
If you can access /tree then, there's no need for nbclassic.
did you try installing the nbextensions in the docker image right after installing the lab extension?

jupyter nbextension install --py widgetsnbextension
jupyter nbextension enable --py widgetsnbextension
jupyter nbextension install --py tensorflow_model_analysis
jupyter nbextension enable --py tensorflow_model_analysis

My tfma setup is probably different from yours.
I used these instructions to install the tfma npm package.
#56 (comment)

ConverJens · 2021-03-15T14:12:52Z

@mwakaba2
That was my understanding but I wanted to verify.

These are the commands I'm currently using:

RUN jupyter contrib nbextension install && \
    jupyter labextension install tensorflow_model_analysis@0.28.0 && \
    jupyter labextension install @jupyter-widgets/jupyterlab-manager@2  && \
    jupyter nbextension install --py --sys-prefix widgetsnbextension && \
    jupyter nbextension enable --py --sys-prefix widgetsnbextension && \
    jupyter nbextension install --py --sys-prefix tensorflow_model_analysis && \
    jupyter nbextension enable --py --sys-prefix tensorflow_model_analysis

along with pip installing the corresponding TFX version first. I've tested 0.26.0, 0.27.0 and 0.28.0 with exactly the same result. I still cannot load the vulcanized_tfma.js file.

@kwlzn @rcrowe-google
I'm not certain the issue that you are mentioning is actually what's causing my problem. I can run my image, notebook and data locally successfully but once I host it, it fails. I can run it locally without the jupyter nbextension steps.

ConverJens · 2021-03-15T14:40:57Z

@atn832 @fhuanming @mwakaba2 @kwlzn @rcrowe-google
So I think I found whats causing this issue: url rewrite in KubeFlow is causing the vulcanized_tfma path to point to the wrong place.

If I check the sources tab in the developer console and right click the vulcanized_tfma.js, and choose "open in new tab" I'm directed to: <kubeflow host>/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js which is incorrect since the notebook lives in: <kubeflow host>/notebook/admin/<notebook server name>. I'm also greated by KubeFlows 'this page doesn't exist' page.

If I manually concatenate these urls I get the actual js file: <kubeflow host>/notebook/admin/<notebook server name>/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js

So it seems that this is actually a path issue. This also perfectly explains why it works when running the image locally but not with the re-written url.

How do we proceed with this?

Bobgy · 2021-09-26T07:05:51Z

I'm starting to reproduce, just verified using a local jupyter lab instance with instructions like #112 (comment). I can see tfma visualization in the notebook.

In this case, vulcanized_tfma.js is downloaded from http://127.0.0.1:8888/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js.

Bobgy · 2021-09-26T08:29:33Z

DISCLAIMER: the urls below are my private Kubeflow instance, so you won't be able to access them.

I reproduced the issue.
However, I think "Please enable Javascript" message is not an actual problem. #112 (comment)
The root cause is that when downloading this file from for example: "https://dev-5-14.endpoints.gongyuan-dev.cloud.goog/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js".
The content it downloads is actually kubeflow central dashboard HTML content (which, as a side effect, when trying to visualize in chrome devtools will show "please enable javascript", because chrome devtools do not enable javascript).
What we expect instead for the response is a minified javascript file for vulcanized_tfma.js.

Bobgy · 2021-09-26T08:30:44Z

I'm getting closer, the correct download url should be "https://dev-5-14.endpoints.gongyuan-dev.cloud.goog/notebook/gongyuan/tfma4/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js".
This one properly returns js content of vulcanized_tfma.js.

Bobgy · 2021-09-26T08:40:33Z

NB_PREFIX env var is "/notebook/gongyuan/tfma4" for me case, so the correct URL is
HOST/NB_PREFIX/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js.
Somehow tfma widget needs to know how to inject NB_PREFIX in there.

Bobgy · 2021-09-26T08:44:10Z

NB_PREFIX is passed to jupyterlab server in https://github.com/kubeflow/kubeflow/blob/f57be7ac6edc03561009eb3ae3f1344e16a05f09/components/example-notebook-servers/jupyter/s6/services.d/jupyterlab/run#L12

--ServerApp.base_url="${NB_PREFIX}"

Bobgy · 2021-09-26T08:55:43Z

That's pretty much what I can figure out based on my knowledge.
@mdreves can you or someone you know help figure out how ServerApp.base_url of jupyterlab can be passed to javascript in a jupyterlab widget? Then we need to fix

model-analysis/tensorflow_model_analysis/notebook/jupyter/js/lib/widget.js

Lines 32 to 34 in e3bb17a

    
           else if (window['isJupyterLab']) { 
        
             templatePath = '/nbextensions/tensorflow_model_analysis/'; 
        
           }

, so that it adds the base_url in front of the '/nbextensions/tensorflow_model_analysis/'.

Bobgy · 2021-09-26T09:01:58Z

Just inspecting on jupyterlab HTML, I found some config data, probably we can directly query for content of the script, but I'm not sure if that's the official way to get the value.

<script id="jupyter-config-data" type="application/json">{"appName": "JupyterLab", "baseUrl": "/notebook/gongyuan/tfma4/" ... </script>

EDIT: my guess was right, https://github.com/jupyterlab/jupyterlab/blob/ce2229e86fc1bf04fdd2ed4dab226a790ad9cd8d/packages/coreutils/src/pageconfig.ts#L25-L34 is code for reading the config data.

PiperOrigin-RevId: 401267422

mdreves · 2021-10-07T18:00:22Z

We think this should be fixed now, but would like someone to try it out. Special thanks to @Bobgy and @zijianjoy.

ConverJens · 2021-10-08T07:23:12Z

@mdreves I can probably test this during next week.

ConverJens · 2021-10-25T11:15:18Z

@mdreves Sorry for the late response! I've had multiple issues that I had to deal with. I'm ready to test this. What do I need to do to install your changes?

jhamet93 · 2021-10-29T13:26:20Z

@mdreves We can try this out as well. Will commit some time to testing the commit.

mdreves · 2021-10-29T17:10:27Z

The 0.35 release will be out shortly if you want to wait for that so you don't need to install from head, though you can also try with the nightly.

ConverJens · 2021-11-03T12:42:28Z

@mdreves
I just tried to test this but when running tfma.view.render_slicing_metrics(eval_result) I got Error displaying widget.
I'm using jupyterlab 3 and I have installed TFMA using the official instructions.
Any idea why I get this?

ConverJens · 2021-11-10T08:41:14Z

@mdreves @jhamet93 @Bobgy @zywind Has anyone else tested this and got this to work in JupyterLab?

zywind · 2021-11-10T15:14:13Z

I've tried it and had the same error with 0.35 release.

mdreves · 2021-11-17T18:43:17Z

My apologies, but I don't have the cycles to work on this and we don't have anyone with expertise in this area at the moment.

Just to clarify, it works except for when you go to display the results using tfma.view.render_slicing_metrics. Is the full error Error displaying widget or is there more information?

zywind · 2021-11-17T20:43:50Z

This is what I see in jupyter lab:

ConverJens · 2021-11-18T07:37:35Z

@mdreves
I see exactly the same as @zywind posted as well.

rcrowe-google · 2021-11-18T20:05:02Z

We'd be very interested if someone can do a PR to fix this.

ConverJens · 2022-01-27T14:21:40Z

@mdreves Do we have any update on this issue? Do you have the time to work on it now?

mdreves · 2022-01-28T20:59:36Z

@embr as FYI since he is the new TL for TFMA.

Unfortunately we still don't have the cycles to work on this and we don't have anyone with expertise in this area at the moment. We would be very happy to see a community contribution in this case.

ConverJens · 2022-04-08T11:38:40Z

@mdreves @embr
Any update on this? Can this be prioritised now?

ConverJens · 2022-04-08T11:39:39Z

@JanetVictorious That's not fixing this at all, you've just just jupyter notebook instead of JupyterLab and this entire issue is about JupyterLab support.

genehwung · 2022-05-18T04:12:01Z

Hopefully someone has the expertise to work on a PR, meanwhile we are actively working on DataFrame API as an alternative to provide pandas DataFrame for easier plotting.

rcrowe-google · 2022-05-18T18:37:52Z

This may or may not help. I recently got the TFMA visualization working in JupyterLab by changing my browser settings. I'm using Chrome, and under Settings > Privacy and security > Security my setting was "Enhanced protection". Changing it to "Standard protection" fixed the issue, which was caused by Chrome stopping what it sees as a CORB request. I was using this code to display the visualization:

from IPython.display import IFrame
from ipywidgets.embed import embed_minimal_html

embed_minimal_html('tfma_slices_overview.html',
                   views=[store.display_tfma_analysis(13, slicing_column='trip_start_hour')],
                   title='Slices Overview')

IFrame(src='tfma_slices_overview.html', width=900, height=600)

I hope that helps!

pindinagesh · 2022-07-07T13:16:17Z

Hi @ConverJens

Can you please confirm on the comment from @rcrowe-google and suggest whether this can be closed. Thank you.

ConverJens mentioned this issue Feb 19, 2021

JupyterLab support? #56

Closed

arghyaganguly self-assigned this Feb 22, 2021

arghyaganguly added the type:bug label Feb 22, 2021

arghyaganguly assigned atn832 and unassigned arghyaganguly Feb 22, 2021

arghyaganguly added the stat:awaiting response label Feb 22, 2021

arghyaganguly added stat:awaiting tensorflower and removed stat:awaiting response labels Feb 22, 2021

arghyaganguly assigned mdreves Feb 22, 2021

kwlzn mentioned this issue Mar 12, 2021

TFMA UI widgets fail to render in nbconvert HTML conversion #116

Open

copybara-service bot pushed a commit that referenced this issue Oct 6, 2021

Read baseUrl in JupyterLab to support TFMA rendering: #112

73299e5

PiperOrigin-RevId: 401267422

This comment was marked as off-topic.

Sign in to view

pindinagesh added stat:awaiting response and removed stat:awaiting tensorflower labels Jul 7, 2022

TFMA not rendering in JupyterLab #112

TFMA not rendering in JupyterLab #112

Comments

ConverJens commented Feb 19, 2021 • edited Loading

System information

Describe the problem

Source code / logs

fhuanming commented Feb 19, 2021

atn832 commented Feb 22, 2021

ConverJens commented Feb 22, 2021

Bobgy commented Feb 25, 2021

ConverJens commented Feb 25, 2021

Bobgy commented Feb 26, 2021

fhuanming commented Mar 1, 2021

atn832 commented Mar 2, 2021

ConverJens commented Mar 2, 2021

ConverJens commented Mar 2, 2021

mwakaba2 commented Mar 9, 2021

rcrowe-google commented Mar 10, 2021 • edited Loading

mwakaba2 commented Mar 10, 2021 • edited Loading

mwakaba2 commented Mar 10, 2021

rcrowe-google commented Mar 11, 2021

kwlzn commented Mar 12, 2021 • edited Loading

ConverJens commented Mar 12, 2021

mwakaba2 commented Mar 12, 2021

ConverJens commented Mar 15, 2021 • edited Loading

ConverJens commented Mar 15, 2021 • edited Loading

Bobgy commented Sep 26, 2021 • edited Loading

Bobgy commented Sep 26, 2021 • edited Loading

Bobgy commented Sep 26, 2021 • edited Loading

Bobgy commented Sep 26, 2021

Bobgy commented Sep 26, 2021

Bobgy commented Sep 26, 2021

Bobgy commented Sep 26, 2021 • edited Loading

mdreves commented Oct 7, 2021

ConverJens commented Oct 8, 2021

ConverJens commented Oct 25, 2021

jhamet93 commented Oct 29, 2021

mdreves commented Oct 29, 2021

ConverJens commented Nov 3, 2021

ConverJens commented Nov 10, 2021

zywind commented Nov 10, 2021

mdreves commented Nov 17, 2021

zywind commented Nov 17, 2021

ConverJens commented Nov 18, 2021

rcrowe-google commented Nov 18, 2021

ConverJens commented Jan 27, 2022

mdreves commented Jan 28, 2022

This comment was marked as off-topic.

ConverJens commented Apr 8, 2022

ConverJens commented Apr 8, 2022

genehwung commented May 18, 2022

rcrowe-google commented May 18, 2022

pindinagesh commented Jul 7, 2022

ConverJens commented Feb 19, 2021 •

edited

Loading

rcrowe-google commented Mar 10, 2021 •

edited

Loading

mwakaba2 commented Mar 10, 2021 •

edited

Loading

kwlzn commented Mar 12, 2021 •

edited

Loading

ConverJens commented Mar 15, 2021 •

edited

Loading

ConverJens commented Mar 15, 2021 •

edited

Loading

Bobgy commented Sep 26, 2021 •

edited

Loading

Bobgy commented Sep 26, 2021 •

edited

Loading

Bobgy commented Sep 26, 2021 •

edited

Loading

Bobgy commented Sep 26, 2021 •

edited

Loading