Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FSP engine task re-syncs should not re-attach to shm #359

Closed
Tracked by #339
goodboy opened this issue Jul 14, 2022 · 0 comments · Fixed by #360
Closed
Tracked by #339

FSP engine task re-syncs should not re-attach to shm #359

goodboy opened this issue Jul 14, 2022 · 0 comments · Fixed by #360
Labels
data-layer real-time and historical data processing and storage fsp financial signal processing

Comments

@goodboy
Copy link
Contributor

goodboy commented Jul 14, 2022

The main error is:

Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/_actor.py", line 193, in _invoke
  File "/home/goodboy/repos/piker/piker/fsp/_engine.py", line 319, in cascade
  File "/home/goodboy/repos/piker/310/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
  File "/home/goodboy/repos/piker/piker/fsp/_engine.py", line 212, in fsp_compute
  File "/home/goodboy/repos/piker/piker/fsp/_volume.py", line 292, in flow_rates
  File "/home/goodboy/repos/piker/piker/fsp/_api.py", line 138, in get_shm
  File "/home/goodboy/repos/piker/piker/data/_sharedmem.py", line 566, in attach_shm_array
  File "/home/goodboy/repos/piker/piker/data/_sharedmem.py", line 556, in attach_shm_array
  File "/usr/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
OSError: [Errno 24] Too many open files: '/xmrbtc.binance.fsp.dolla_vlm.chart.15d82439-b7a1-40f0-8e24-9f67e6103523'

The source issue for this #332 but this will likely be the right workaround for that until someone can hack that issue directly in the stdlib.

the main issue is that when the underlying fsp function is run it will re-attach to any dependee shms declared in the @fsp decorator wherein the underlying Fsp.get_shm() recalls attach_shm() instead of using a cached version..

funny enough, there's already a comment about doing this 😂
@goodboy goodboy added data-layer real-time and historical data processing and storage fsp financial signal processing labels Jul 14, 2022
goodboy added a commit that referenced this issue Jul 19, 2022
@goodboy
Cache fsp ShmArrays where possible
2aeda3b 
Minimize calling `.data._shmarray.attach_shm_array()` as much as is
possible to avoid the crash from #332. This is the suggested hack from
issue #359.

Resolves #359
goodboy added a commit that referenced this issue Jul 19, 2022
@goodboy
Cache fsp ShmArrays where possible
e0491cf 
Minimize calling `.data._shmarray.attach_shm_array()` as much as is
possible to avoid the crash from #332. This is the suggested hack from
issue #359.

Resolves #359
@goodboy goodboy mentioned this issue Jul 19, 2022
Merged
This was referenced Oct 15, 2022
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
data-layer real-time and historical data processing and storage fsp financial signal processing
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fsp shm caching pikers/piker
1 participant
@goodboy