Fix leak of iopub object in activity monitoring #3424
Conversation
After analyzing various leaked items when running either Notebook or Jupyter Kernel Gateway, one item that recurred across each kernel startup and shutdown sequence was the a zmq socket instance associated with the iopub port. The leaked instance occurs because the activity monitor and normal port creation logic both call `create_iopub`. Although this ens up creating a _wrapper_ around the same port, the object (i.e., wrapper) created by the activity monitor is leaked. By setting the kernel manager's `_activity_stream` member to `None` when the kernel is shutdown, cleanup of the iopub wrapper object will take place.
PR jupyter-server#279 introduced some fixes to address file descriptor leaks - one of which was to shutdown the communication port. Since the kernel launcher listening on the other side may have already terminated, the shutdown method could throw an exception. This change catches, logs, then ignores such exceptions. While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
|
I don't immediately see why this would result in a leak; the kernel object itself is deleted shortly afterwards (the What am I missing? |
|
@takluyver - thanks for the response. I agree that its quite non-obvious. However, w/o this statement, the iopub object wrapper created here will not be garbage collected - with a new (leaked) instance persisted across each subsequent kernel cycle. I found this and the two PRs (360, 361) submitted to jupyter_client via use of However, since this leak was found prior to the resolution to the IOLoopKernelManager leak (via PR 361) then I suppose the change may not be required - although, in my opinion, is good practice. I didn't go and back out this change after breaking the circular references that were preventing the leak in 361. |
|
Hmm - I just commented out this change (and running with all other fixes in place) and see the iopub object and the kernel manager instance leaks return. I don't see an obvious circular reference introduced by |
While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
|
@takluyver - is there anything else you need for this? |
|
No, I don't think so. I'll close and reopen it to rerun the tests. |
|
Thank you Thomas! |
After analyzing various leaked items when running either Notebook or
Jupyter Kernel Gateway, one item that recurred across each kernel
startup and shutdown sequence was the a zmq socket instance associated
with the iopub port. The leaked instance occurs because the activity
monitor and normal port creation logic both call
create_iopub.Although this ends up creating a wrapper around the same port, the
object (i.e., wrapper) created by the activity monitor is leaked. By
setting the kernel manager's
_activity_streammember toNonewhenthe kernel is shutdown, cleanup of the iopub wrapper object will take
place.