Initial hack to get dask distributed working #1083
Conversation
| if type_ is not None: | ||
| header['type'] = type_ | ||
| array = distributed.protocol.deserialize(header, frames) | ||
| return CharToStringArray(array) |
There was a problem hiding this comment.
Any objection if I change this (and the other similar functions) to use the sub-types list solution?
There was a problem hiding this comment.
That would be fine, but these classes (other than LazilyIndexedArray) are not going to be layered multiple times. So I wouldn't worry too much about that until we settle on a general solution.
|
It occurs to me after reading your DataStore API issue that we could also just implement getstate/setstate methods on all of the wrapping classes (like LazilyIndexedArray) that thought about how to serialize their contained arrays. You wouldn't have to use dask's mechanism. You would still have to figure out how to effectively serialize and deserialize any contained object, which is the same problem we're solving at dask/distributed#620 , but could stick to the pickle protocol if you wanted to stay with more general and established protocols. You are reaching down a little bit in the object tree further than was intended, but that might be a more palatable long term solution for xarray? |
|
Not that I'm recommending this, just that it is another option. |
If I understand correctly, I could even just go to the bottom level, and make This does seem a little more palatable in the long term, especially if we build some more general machinery that DataStores can plug into (added into #1087) |
|
Closing in favor of #1095 |
WIP WIP WIP
CC
@mrocklin
@pwolfram
xref #798