Distributed hello world fails when using jemalloc

[bluenote10]

Observation made in relation to #1179.

Running a local distributed executor like in the "hello world" example crashes if the allocator is jemalloc.

Starting an IPython shell via LD_PRELOAD=/usr/lib/libjemalloc.so.1 ipython and running the following code

from dask.distributed import Client
client = Client()

def square(x):
    return x ** 2

def neg(x):
    return -x

A = client.map(square, range(10))
B = client.map(neg, A)
total = client.submit(sum, B)
print(total.result())

crashes with ValueError: Worker not started

The code works fine using glibc.

[jreback]

xref pandas-dev/pandas#16727

[xhochy]

@jreback no, they don't seem to be related here.

Sadly I cannot reproduce this problem on my machines here (neither OSX nor Debian 7+). It seems that there is some relation to the old Ubuntu 12.04 image @bluenote10 is running on which ships with jemalloc 2.2.x and glibc 2.13.

[pitrou]

I cannot reproduce on Ubuntu 16.04 either. In any case, this doesn't seem to be a Distributed bug, so I'm inclined to close this issue unless you have significant reason to believe Distributed is involved.

[bluenote10]

I'm not sure. Similar to the other issue it seems to affect the networking. Maybe you can make more sense of the full traceback:

tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.py", line 173, in _start_worker
    yield w._start()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 135, in _start
    response = yield self.instantiate()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 189, in instantiate
    self.process.start())
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 300, in start
    yield self._wait_until_running()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.py", line 385, in _wait_until_running
    raise ValueError("Worker not started")
ValueError: Worker not started
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9a5cb41bb097> in <module>()
      1 from dask.distributed import Client
----> 2 client = Client()
      3 
      4 def square(x):
      5     return x ** 2

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, start, asynchronous, **kwargs)
    471                                      io_loop=self.loop)
    472 
--> 473         self.start(timeout=timeout, asynchronous=asynchronous)
    474 
    475         from distributed.channels import ChannelClient

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in start(self, asynchronous, **kwargs)
    570             self._started = self._start(**kwargs)
    571         else:
--> 572             sync(self.loop, self._start, **kwargs)
    573         self.status = 'running'
    574 

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/utils.pyc in sync(loop, func, *args, **kwargs)
    232         e.wait(1000000)
    233     if error[0]:
--> 234         six.reraise(*error[0])
    235     else:
    236         return result[0]

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/utils.pyc in f()
    221                 raise RuntimeError("sync() called from thread of running loop")
    222             yield gen.moment
--> 223             result[0] = yield make_coro()
    224         except Exception as exc:
    225             logger.exception(exc)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in _start(self, timeout, **kwargs)
    614                 self.cluster = LocalCluster(loop=self.loop, start=False,
    615                                             **self._startup_kwargs)
--> 616                 yield self.cluster._start()
    617             except (OSError, socket.error) as e:
    618                 if e.errno != errno.EADDRINUSE:

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start(self, ip)
    146         yield self._start_all_workers(
    147             self.n_workers, ncores=self.threads_per_worker,
--> 148             services=self.worker_services, **self.worker_kwargs)
    149 
    150         self.status = 'running'

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start_all_workers(self, n_workers, **kwargs)
    152     @gen.coroutine
    153     def _start_all_workers(self, n_workers, **kwargs):
--> 154         yield [self._start_worker(**kwargs) for i in range(n_workers)]
    155 
    156     @gen.coroutine

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in callback(f)
    826             for f in children:
    827                 try:
--> 828                     result_list.append(f.result())
    829                 except Exception as e:
    830                     if future.done():

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/deploy/local.pyc in _start_worker(self, port, processes, death_timeout, **kwargs)
    171               death_timeout=death_timeout,
    172               silence_logs=self.silence_logs, **kwargs)
--> 173         yield w._start()
    174 
    175         self.workers.append(w)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in _start(self, addr_or_port)
    133 
    134         logger.info('        Start Nanny at: %r', self.address)
--> 135         response = yield self.instantiate()
    136         if response == 'OK':
    137             assert self.worker_address

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in instantiate(self, comm)
    187             try:
    188                 yield gen.with_timeout(timedelta(seconds=self.death_timeout),
--> 189                                        self.process.start())
    190             except gen.TimeoutError:
    191                 yield self._close(timeout=self.death_timeout)

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1061                     if exc_info is not None:
   1062                         try:
-> 1063                             yielded = self.gen.throw(*exc_info)
   1064                         finally:
   1065                             # Break up a reference to itself

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in start(self)
    298         yield self.process.start()
    299         if self.status == 'starting':
--> 300             yield self._wait_until_running()
    301 
    302     def _on_exit(self, proc):

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1053 
   1054                     try:
-> 1055                         value = future.result()
   1056                     except Exception:
   1057                         self.had_exception = True

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.pyc in result(self, timeout)
    236         if self._exc_info is not None:
    237             try:
--> 238                 raise_exc_info(self._exc_info)
    239             finally:
    240                 self = None

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.pyc in run(self)
   1067                             exc_info = None
   1068                     else:
-> 1069                         yielded = self.gen.send(value)
   1070 
   1071                     if stack_context._state.contexts is not orig_stack_contexts:

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/nanny.pyc in _wait_until_running(self)
    383         while True:
    384             if self.status != 'starting':
--> 385                 raise ValueError("Worker not started")
    386             try:
    387                 msg = self.init_result_q.get_nowait()

ValueError: Worker not started

[pitrou]

The fact that networking may be affected doesn't imply that Distributed is the culprit. Distributed is pure Python code and does not depend in any particular way on the underlying C malloc implementation. According to this message there might be a bad interaction between specific glibc and jemalloc versions.

I might add: why are you using jemalloc? Did you get specific performance improvements using it?

[bluenote10]

Result of valgrind leading to the traceback

==23375== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23375==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23375==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x544CBB: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==  Address 0xffeffb998 is on thread 1's stack
==23375== 
==23854== Thread 13:
==23854== Invalid read of size 8
==23854==    at 0x4E39E01: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== Invalid read of size 8
==23854==    at 0x4E39F65: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== Invalid read of size 4
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==  Address 0x4ccccccd048d2cb0 is not stack'd, malloc'd or (recently) free'd
==23854== 
==23854== 
==23854== Process terminating with default action of signal 11 (SIGSEGV)
==23854==  General Protection Fault
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23854==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23854== 
==23854== HEAP SUMMARY:
==23854==     in use at exit: 240 bytes in 3 blocks
==23854==   total heap usage: 3 allocs, 0 frees, 240 bytes allocated
==23854== 
==23856== Thread 14:
==23856== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23856==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23856==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x3a843518 is on thread 14's stack
==23856== 
==23860== Thread 15:
==23860== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23860==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23860==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23860==  Address 0x3b7fd518 is on thread 15's stack
==23860== 
==23864== Thread 16:
==23864== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23864==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23864==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23864==  Address 0x3c7fd518 is on thread 16's stack
==23864== 
==23867== Thread 17:
==23867== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23867==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23867==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23867==  Address 0x3d7fd518 is on thread 17's stack
==23867== 
==23856== Invalid read of size 8
==23856==    at 0x4E39E01: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== Invalid read of size 8
==23856==    at 0x4E39F65: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x35c00000 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== Invalid read of size 4
==23856==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==  Address 0x4ccccccd078d2cb0 is not stack'd, malloc'd or (recently) free'd
==23856== 
==23856== 
==23856== Process terminating with default action of signal 11 (SIGSEGV)
==23856==  General Protection Fault
==23856==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23856==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23856==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23856==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)
==23856==    by 0x49F3C8: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5376F1: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x51B6A7: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x537635: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23856== 
==23856== HEAP SUMMARY:
==23856==     in use at exit: 240 bytes in 3 blocks
==23856==   total heap usage: 3 allocs, 0 frees, 240 bytes allocated
==23856== 
==23870== Thread 18:
==23870== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23870==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23870==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23870==  Address 0x3e7fd518 is on thread 18's stack
==23870== 
==23375== Thread 12:
==23375== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23375==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23375==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23375==  Address 0x2b3fec88 is on thread 12's stack
==23375== 
==23874== Thread 20:
==23874== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23874==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23874==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23874==  Address 0x407fd518 is on thread 20's stack
==23874== 
==23877== Thread 19:
==23877== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==23877==    at 0x62B992A: epoll_ctl (syscall-template.S:82)
==23877==    by 0x544B9B: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x544D13: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x536F93: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x51D1D1: ??? (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x4C2645: PyObject_Call (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x538C1F: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x53E2AF: PyEval_EvalCodeEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==    by 0x5374A7: PyEval_EvalFrameEx (in /home/PHI-TPS/fkeller/.virtualenvs/dask/bin/python)
==23877==  Address 0x3f7fd518 is on thread 19's stack
==23877== 
==23854== LEAK SUMMARY:
==23854==    definitely lost: 0 bytes in 0 blocks
==23854==    indirectly lost: 0 bytes in 0 blocks
==23854==      possibly lost: 0 bytes in 0 blocks
==23854==    still reachable: 240 bytes in 3 blocks
==23854==         suppressed: 0 bytes in 0 blocks
==23854== Reachable blocks (those to which a pointer was found) are not shown.
==23854== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23854== 
==23854== For counts of detected and suppressed errors, rerun with: -v
==23854== Use --track-origins=yes to see where uninitialised values come from
==23854== ERROR SUMMARY: 9 errors from 4 contexts (suppressed: 0 from 0)
==23856== LEAK SUMMARY:
==23856==    definitely lost: 0 bytes in 0 blocks
==23856==    indirectly lost: 0 bytes in 0 blocks
==23856==      possibly lost: 0 bytes in 0 blocks
==23856==    still reachable: 240 bytes in 3 blocks
==23856==         suppressed: 0 bytes in 0 blocks
==23856== Reachable blocks (those to which a pointer was found) are not shown.
==23856== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23856== 
==23856== For counts of detected and suppressed errors, rerun with: -v
==23856== Use --track-origins=yes to see where uninitialised values come from
==23856== ERROR SUMMARY: 17 errors from 5 contexts (suppressed: 0 from 0)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9ae027076550> in <module>()
      1 from dask.distributed import Client
----> 2 client = Client()

/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/client.pyc in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, start, asynchronous, **kwargs)
    471                                      io_loop=self.loop)
    472 
--> 473         self.start(timeout=timeout, asynchronous=asynchronous)
    474 
    475         from distributed.channels import ChannelClient

[bluenote10]

I had less issues resulting from memory fragmentation with jemalloc, but I should be able to use glibc as well.

[xhochy]

@pitrou The arrow issues were solely a build environment issue that surfaced while loading a library. Once the library is successfully loaded, this glibc bug is not triggered anymore.

We're using jemalloc in general (note: I work at the same place as @bluenote10 ;) ) as it has less memory fragmentation as glibc and has a better multithreaded performance. In Apache Arrow (different use case then in this issue here), we also use it as it can provide aligned memory (re-)allocation, this enables us to use faster numeric CPU instructions.

[pitrou]

Ok. Still, I don't know what to do with this issue. Using a different memory allocator shouldn't mess network communications implemented in pure Python, unless there's something seriously wrong in low-level routines (I mean routines implemented in C either inside Python itself or inside system libraries and/or third-party C libraries).

As for the Valgrind output, the following seems fishy. But a Valgrind or glibc expert would have to dig in:

==23854== Process terminating with default action of signal 11 (SIGSEGV)
==23854==  General Protection Fault
==23854==    at 0x4E39F86: free (in /usr/lib/libjemalloc.so.1)
==23854==    by 0x50583B8: get_cached_stack (allocatestack.c:249)
==23854==    by 0x50583B8: allocate_stack (allocatestack.c:483)
==23854==    by 0x50583B8: pthread_create@@GLIBC_2.2.5 (pthread_create.c:462)

[bluenote10]

Sure we can close it, if there is nothing that can be done. Just wanted to keep you posted on the issue.