OS X: Shm issue `OSError: [Errno 63] File name too long:` #306

torresnelson · 2022-04-18T14:11:07Z

Using this command piker -l info -b binance chart btcusdt.binanc I'm getting this error:

Apr 18 10:50:19 (brokerd.binance, 15316, piker.data.feed.allocate_persistent_feed)) [WARNING] piker.data.feed feed.py:383 btcusdt.binance feed task terminated
Apr 18 10:50:19 (brokerd.binance, 15316, piker.brokers.binance.stream_quotes)) [INFO] tractor.trionics._mngrs _mngrs.py:215 De-allocating resource for (4934515248, ())
Remote context error for ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b'):c2015eba-ca1f-4225-bbf2-82e957ccadb8:
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_actor.py", line 224, in _invoke
    raise ContextCancelled(
tractor._exceptions.ContextCancelled: _setup_persistent_brokerd was remotely cancelled by its caller ('pikerd', 'a82b89dc-0af0-4bbf-a18d-cedf6b129fa1')

Nursery for ('pikerd', 'a82b89dc-0af0-4bbf-a18d-cedf6b129fa1') errored with [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1',
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_supervise.py", line 327, in _open_and_supervise_one_cancels_all_nursery
    yield anursery
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_supervise.py", line 494, in open_nursery
    yield anursery
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 178, in open_pikerd
    yield _services
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 279, in maybe_open_pikerd
    yield None
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
Apr 18 10:50:19 (brokerd.binance, 15316, tractor._actor.Actor._async_main)) [INFO] tractor _actor.py:1209 Runtime nursery complete
Apr 18 10:50:19 (brokerd.binance, 15316, tractor._actor.Actor._async_main)) [INFO] tractor _actor.py:1213 Closing all actor lifetime contexts
Apr 18 10:50:19 (brokerd.binance, 15316, no task context)) [INFO] tractor._entry _entry.py:123 Actor ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b') terminated
Cancelling existing result waiter task for ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b')
Root actor crashed:
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_root.py", line 204, in open_root_actor
    yield actor
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 178, in open_pikerd
    yield _services
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 279, in maybe_open_pikerd
    yield None
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
< Exiting ui._fsp.start_fsp_displays, total time: 4649.5217 ms

The text was updated successfully, but these errors were encountered:

goodboy · 2022-04-18T14:12:53Z

OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'

Very interesting. I guess we could try truncating it on OS X?
I'm not sure about how peeps normally solve this issue tho i have definitely seen it before when working such machines.

this is actually something that needs to get dug into anyway - better organizing shm files on the system to be,

easier to wipe when things go wrong and files are left stale
clearer to the user which array / entry maps to which app/service.

torresnelson · 2022-04-18T15:04:45Z

Truncating it at mk_fsp_shm_key() throws the same error: OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.c84'

goodboy · 2022-04-30T17:42:12Z

I've also seen a new issue with "too many open files" as an OSError coming from this code.

We'll need to dig in at some point but for this simple case what is the max "file name length" on OS X?

goodboy · 2022-06-04T18:22:55Z

@torresnelson yeah not sure what to say (btw) bc i don't run OS X, so you'll have to do some tinkering on your own or write a test we can stick in CI.

goodboy added bug guille broke it prolly data-layer real-time and historical data processing and storage labels Apr 18, 2022

goodboy changed the title ~~Error at initialization on macos~~ OS X: Shm issue OSError: [Errno 63] File name too long: Oct 2, 2022

This was referenced Oct 15, 2022

Shared memory array API and optional tight integration with numpy goodboy/tractor#338

Open

Share memory (array) sub-system API and optional tight integration with numpy goodboy/tractor#339

Open

jaredgoldman mentioned this issue Oct 20, 2022

OS X: shared memory file name length limitation causing error goodboy/tractor#340

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OS X: Shm issue `OSError: [Errno 63] File name too long:` #306

OS X: Shm issue `OSError: [Errno 63] File name too long:` #306

torresnelson commented Apr 18, 2022 •

edited

Loading

goodboy commented Apr 18, 2022 •

edited

Loading

torresnelson commented Apr 18, 2022 •

edited

Loading

goodboy commented Apr 30, 2022

goodboy commented Jun 4, 2022

OS X: Shm issue OSError: [Errno 63] File name too long: #306

OS X: Shm issue OSError: [Errno 63] File name too long: #306

Comments

torresnelson commented Apr 18, 2022 • edited Loading

goodboy commented Apr 18, 2022 • edited Loading

torresnelson commented Apr 18, 2022 • edited Loading

goodboy commented Apr 30, 2022

goodboy commented Jun 4, 2022

OS X: Shm issue `OSError: [Errno 63] File name too long:` #306

OS X: Shm issue `OSError: [Errno 63] File name too long:` #306

torresnelson commented Apr 18, 2022 •

edited

Loading

goodboy commented Apr 18, 2022 •

edited

Loading

torresnelson commented Apr 18, 2022 •

edited

Loading