Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate pickle 5 protocol for extending design space for zero-copy #189

Closed
sighingnow opened this issue Feb 3, 2021 · 1 comment · Fixed by #213
Closed

Investigate pickle 5 protocol for extending design space for zero-copy #189

sighingnow opened this issue Feb 3, 2021 · 1 comment · Fixed by #213
Labels
component:python Issues about vineyard's python SDK component:vineyardd Issues about vineyardd proposal Enhancement proposals for vineyard.

Comments

@sighingnow
Copy link
Member

sighingnow commented Feb 3, 2021

Describe your problem

I just know that apache-arrow will deprecated the serialization implementation and move to pickle5 (ARROW-11463).

We need to check if that is suitable for vineyard.

**Notes about pickle 5

This PEP aims to make pickle usable in a way where large data is handled as a separate stream of zero-copy buffers, letting the application handle those buffers optimally.

Pretty like what we have in vineyard: an object consists of a metadata and a set of blobs...

Additional context

See also: #184
And:

@sighingnow sighingnow added component:python Issues about vineyard's python SDK component:vineyardd Issues about vineyardd proposal Enhancement proposals for vineyard. labels Feb 3, 2021
@sighingnow
Copy link
Member Author

sighingnow commented Feb 3, 2021

Serialization using the new pickle in mars: https://github.com/mars-project/mars/pull/1937/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:python Issues about vineyard's python SDK component:vineyardd Issues about vineyardd proposal Enhancement proposals for vineyard.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant