Idea💡 Use Cloud Native K8S Jupyter Operator

[EugeneTorap]

Hi everyone👋 I'm contributing to a Elastic Jupyter Operator. This is open-source project written in Golang and based on Kubernetes and Enterprise Gateway.

This operator is to allow:

• Configure kernels on the fly.
• Support adding/removing kernel specs dynamically.
• Remote Jupyter kernel execution with custom configuration.
• Support custom kernel image, command, resource requirements and so on.

Read more about the project Operator Doc and Kernel CRD Doc

@kevin-bates @telamonian You can easy install the jupyter operator on minikube or k8s cluster

Installation commands

git clone https://github.com/tkestack/elastic-jupyter-operator.git
cd elastic-jupyter-operator
kubectl apply -f ./hack/enterprise_gateway/prepare.yaml
make deploy
kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupyternotebook.yaml
kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupytergateway.yaml
kubectl port-forward deploy/jupyternotebook-elastic 8888:8888

Can jupyter org create a new github repository for example jupyter-operator and use this codebase?
Jupyter org has a lot of contributors and I think we can make the best jupyter operator.

[kevin-bates]

Hi @EugeneTorap - this sounds very interesting!

I haven't had a chance to wrap my head around all of this, but I wanted to point out that EG will be undergoing some fairly significant changes when it moves its process proxies to kernel provisioners. This is because the concept of process proxies is now part of the jupyter framework (at least it's available to any applications that use jupyter_client to launch kernels). As a result, EG will eventually NOT deliver its process-proxies and kernelspecs, but, instead, those kinds of things will be available via a remote provisioners repository (or multiple repositories). This is why the current EG (with process-proxies) requires jupyter_client < 7 and is incompatible with current jupyter_client releases (>= 7).

I mention this because I notice in one of the sublinks that you're also building another process-proxy, and I wonder if this effort should be targeting the "new world" - one based on kernel provisioners rather than process proxies. You'll find a significant similarity between process-proxies and kernel provisioners (since the latter is essentially conceived from the former).

If you (and others) would like to talk about this, I'd be happy to set up an online meeting where we could hash things out a bit.