Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more parallelism backends? #2

Open
wint3ria opened this issue Oct 10, 2022 · 1 comment
Open

Support more parallelism backends? #2

wint3ria opened this issue Oct 10, 2022 · 1 comment

Comments

@wint3ria
Copy link

I see punpy uses builtin multiprocessing pools to parallelise its computations. Would it be possible to support other parallelism backends? Assuming a pure functional measurement function, it would be theoretically possible to launch it anywhere?

I am thinking about Dask RN. Supporting such a framework would allow to easily launch computations on many machines, which could be helpful when the measurement function is expensive to compute.

I believe it would be feasible to abstract the computation backend, so that users may use whatever they need/want, instead of importing a non-builtin parallelism backend as a new dependency for Punpy. multiprocessing.Pool would then be the default backend.

@wint3ria
Copy link
Author

wint3ria commented Oct 20, 2022

It would be interesting to provide an access to the parallelism backend from the measurement function, so that this later can launch parallel subproblems on the same backend, and resume when these terminate. This would require an asynchronous implementation however, which may be more complicated to do for the time being.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant