Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raise a more informative error when pickling #602

Closed
shoyer opened this issue Nov 6, 2016 · 5 comments
Closed

Raise a more informative error when pickling #602

shoyer opened this issue Nov 6, 2016 · 5 comments

Comments

@shoyer
Copy link
Contributor

shoyer commented Nov 6, 2016

pickle is Python's default serialization method, so lots of libraries use it.

netCDF4 does not support pickle, which is actually pretty reasonable -- it's not obvious how to (de)serialize a netCDF file (e.g., just with the filename, or with a complete image of the file?).

But, we should raise more informative errors when pickling Dataset/Variable objects. For example, netCDF4 currently produces hard to interpret error messages (RuntimeError: NetCDF: Not a valid ID) only after attempted access to data from a pickled/unpickled netCDF4.Variable: dask/distributed#629

CC @mrocklin @pwolfram

@jswhit
Copy link
Collaborator

jswhit commented Nov 7, 2016

How about adding a __reduce__ method with:

def __reduce__(self):         
    raise NotImplementedError('Dataset is not picklable')

@shoyer
Copy link
Contributor Author

shoyer commented Nov 7, 2016

Yes, that would be great!

@jswhit
Copy link
Collaborator

jswhit commented Nov 7, 2016

see pull request #604

jswhit added a commit that referenced this issue Nov 8, 2016
raise NotImplementedError when trying to pickle extension objects (issue #602)
@jswhit
Copy link
Collaborator

jswhit commented Nov 8, 2016

pull request #604 merged

@jswhit jswhit closed this as completed Nov 8, 2016
@shoyer
Copy link
Contributor Author

shoyer commented Nov 8, 2016

Thanks Jeff!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants