How to share kernel with nbconvert Python API

[pplonski]

Hi!

I would like to start a kernel with Kernel Manager and share it between nbconvert executions. Below is the code that creates two notebooks (one with variable definition, and one with variable print). I would like to execute the first notebook and then the second in the same kernel with the ExecutePreprocessor API. Is it possible?

import nbformat 
from jupyter_client import MultiKernelManager 
from nbconvert.preprocessors import ExecutePreprocessor

mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel")
km = mkm.get_kernel("some-kernel")

nb1 = nbformat.v4.new_notebook()
nb2 = nbformat.v4.new_notebook()

nb1["cells"] = [nbformat.v4.new_code_cell("a = 13")]
nb2["cells"] = [nbformat.v4.new_code_cell("print(a)")]

ep = ExecutePreprocessor()
ep.preprocess(nb1, km=km)
ep.preprocess(nb2, km=km)

print(nb1.cells)
print(nb2.cells)

I've asked similar question on nbclient repo: jupyter/nbclient#238

[pplonski]

When I run the above code I got the error:

Traceback (most recent call last):
  File "share.py", line 16, in <module>
    ep.preprocess(nb1, km=km)
  File "/home/piotr/sandbox/python-dashboard/pdenv/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 86, in preprocess
    info_msg = self.wait_for_reply(self.kc.kernel_info())
AttributeError: 'NoneType' object has no attribute 'kernel_info'

From what I've checked in the code in ExecutePreprocessor looks like KernelClient is not initialized in NotebookClient (from nbclient package):

nbconvert/nbconvert/preprocessors/execute.py

Lines 85 to 89 in 7a703b9

	with self.setup_kernel():
	info_msg = self.wait_for_reply(self.kc.kernel_info())
	self.nb.metadata["language_info"] = info_msg["content"]["language_info"]
	for index, cell in enumerate(self.nb.cells):
	self.preprocess_cell(cell, resources, index)

@davidbrochart is it the problem with nbclient? The ExecuteProcessor derives from NotebookClient. I was using the newest version (0.6.5) of nbclient when running the code.

[davidbrochart]

Yes, I forgot to do the same change in nbclient's blocking setup_kernel 😄
I just opened jupyter/nbclient#241.

[pplonski]

@davidbrochart thank you!

[pplonski]

Closing the issue. After @davidbrochart fix and nbclient update to 0.6.6 the example code is working. Thank you!