Notebook error when calling mutliprocessing

[bhilbert4]

This is actually an error related to NIRCam_Gsim, but I wanted to record it here. When running a notebook that creates a WFSS simulation, I'm getting lots of errors like the traceback below. After some investigation, I think this is due to a problem with calling multiprocessing inside of a notebook. (Of course, we're not calling it directly in the notebook, but inside NIRCam_Gsim, which is being imported by wfss_simulator, which is what is being called in the notebook.) Either way, the same commands work fine when run as a python script, so I do believe this issue is related to the notebook itself.

The closest I've come to finding out what is going on is captured here: jupyter/jupyter_client#541
But again, in our case we're not calling multiprocessing directly within the notebook.

Tagging @NorPirzkal so that he is aware of the issue.

[E 13:51:40.961 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x7fad7ec2fea0>)
    Traceback (most recent call last):
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/tornado/ioloop.py", line 741, in _run_callback
        ret = callback()
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 535, in <lambda>
        self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 451, in _handle_events
        self._handle_recv()
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 434, in _run_callback
        callback(*args, **kwargs)
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/notebook/services/kernels/kernelmanager.py", line 418, in record_activity
        msg = session.deserialize(fed_msg_list)
      File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/jupyter_client/session.py", line 921, in deserialize
        raise ValueError("Duplicate Signature: %r" % signature)
    ValueError: Duplicate Signature: b'69c3e5d9b78d5e0061456622f2aea1347c2d7775518a4a192957ceb066177958'

[eas342]

Ah, I got this error today too. Glad you know generally where it's coming from.

[NorPirzkal]

Hi Yes, I think it is jupyter related. I have not had this problem on my own machine, which is weird since I use it a lot. N On Jan 27, 2021, at 3:31 PM, Bryan Hilbert <notifications@github.com<mailto:notifications@github.com>> wrote: This is actually an error related to NIRCam_Gsim, but I wanted to record it here. When running a notebook that creates a WFSS simulation, I'm getting lots of errors like the traceback below. After some investigation, I think this is due to a problem with calling multiprocessing inside of a notebook. (Of course, we're not calling it directly in the notebook, but inside NIRCam_Gsim, which is being imported by wfss_simulator, which is what is being called in the notebook.) Either way, the same commands work fine when run as a python script, so I do believe this issue is related to the notebook itself. The closest I've come to finding out what is going on is captured here: jupyter/jupyter_client#541<jupyter/jupyter_client#541> But again, in our case we're not calling multiprocessing directly within the notebook. Tagging @NorPirzkal<https://github.com/NorPirzkal> so that he is aware of the issue. [E 13:51:40.961 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x7fad7ec2fea0>) Traceback (most recent call last): File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 535, in <lambda> self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0)) File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 451, in _handle_events self._handle_recv() File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv self._run_callback(callback, msg) File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 434, in _run_callback callback(*args, **kwargs) File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/notebook/services/kernels/kernelmanager.py", line 418, in record_activity msg = session.deserialize(fed_msg_list) File "/Users/hilbert/miniconda3/envs/mirage-3.8/lib/python3.6/site-packages/jupyter_client/session.py", line 921, in deserialize raise ValueError("Duplicate Signature: %r" % signature) ValueError: Duplicate Signature: b'69c3e5d9b78d5e0061456622f2aea1347c2d7775518a4a192957ceb066177958' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#614>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJKB6MNACHD3U7NY2VVKFETS4BZZRANCNFSM4WV4NINA>.