Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use _unstack_once for valid dask and sparse versions #5315

Merged
merged 26 commits into from
May 17, 2021
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8497d91
optional import versions to pycompat
Illviljan May 15, 2021
2c11948
Update indexing.py
Illviljan May 15, 2021
aeea5cc
move dtypes to avoid circular import
Illviljan May 15, 2021
8fa535c
Update pycompat.py
Illviljan May 15, 2021
ec6c301
faster unstacking
Illviljan May 15, 2021
3f2784d
Update dataset.py
Illviljan May 15, 2021
de15c39
Update dataset.py
Illviljan May 15, 2021
682922f
have to check that the array type is in use
Illviljan May 15, 2021
dba4693
Update dataset.py
Illviljan May 15, 2021
bb74dc9
sparse arg requires the slow path?
Illviljan May 15, 2021
8f15a28
cupy.__version__
Illviljan May 15, 2021
cd39b67
Merge branch 'Illviljan-version_to_pycompat' into Illviljan-faster_un…
Illviljan May 15, 2021
c2ce0cf
test pint_array_type
Illviljan May 15, 2021
9992824
Update dataset.py
Illviljan May 15, 2021
95fc10e
Add pint check in pycompat
Illviljan May 15, 2021
2597619
Update pycompat.py
Illviljan May 15, 2021
cf80360
revert pint test
Illviljan May 15, 2021
b41c113
import whole dtypes
Illviljan May 16, 2021
df425a4
Merge branch 'Illviljan-version_to_pycompat' into Illviljan-faster_un…
Illviljan May 16, 2021
a31754b
Test turning off the ndarray check
Illviljan May 16, 2021
207609b
sparse and pint doesn't work. Switch to a restrictive if
Illviljan May 16, 2021
7bb5aa5
Update dataset.py
Illviljan May 16, 2021
0d08e35
Merge branch 'master' into Illviljan-faster_unstacking
Illviljan May 17, 2021
d5b31ae
Add back some comments and add some relevant issue links
Illviljan May 17, 2021
35db4a8
lint
Illviljan May 17, 2021
7ee325d
Update dataset.py
Illviljan May 17, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 17 additions & 29 deletions xarray/core/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@
)
from .missing import get_clean_interp_index
from .options import OPTIONS, _get_keep_attrs
from .pycompat import is_duck_dask_array, sparse_array_type
from .pycompat import dask_version, is_duck_dask_array
from .utils import (
Default,
Frozen,
Expand Down Expand Up @@ -4028,36 +4028,24 @@ def unstack(

result = self.copy(deep=False)
for dim in dims:

if (
# Dask arrays don't support assignment by index, which the fast unstack
# function requires.
# https://github.com/pydata/xarray/pull/4746#issuecomment-753282125
any(is_duck_dask_array(v.data) for v in self.variables.values())
# Sparse doesn't currently support (though we could special-case
# it)
# https://github.com/pydata/sparse/issues/422
or any(
isinstance(v.data, sparse_array_type)
for v in self.variables.values()
)
or sparse
# Until https://github.com/pydata/xarray/pull/4751 is resolved,
# we check explicitly whether it's a numpy array. Once that is
# resolved, explicitly exclude pint arrays.
# # pint doesn't implement `np.full_like` in a way that's
# # currently compatible.
# # https://github.com/pydata/xarray/pull/4746#issuecomment-753425173
Illviljan marked this conversation as resolved.
Show resolved Hide resolved
# # or any(
# # isinstance(v.data, pint_array_type) for v in self.variables.values()
# # )
or any(
not isinstance(v.data, np.ndarray) for v in self.variables.values()
)
if not sparse and all(
# Dask arrays recently supports assignment by index,
# https://github.com/dask/dask/pull/7393
dask_version >= "2021.04.0" and is_duck_dask_array(v.data)
# Numpy arrays handles the fast path:
or isinstance(v.data, np.ndarray)
for v in self.variables.values()
):
result = result._unstack_full_reindex(dim, fill_value, sparse)
else:
# Fast unstacking path:
result = result._unstack_once(dim, fill_value)
else:
# Slower unstacking path, examples of array types that
# currently has to use this path:
# * sparse doesn't support item assigment,
# https://github.com/pydata/sparse/issues/114
# * pint has some circular import issues,
# https://github.com/pydata/xarray/pull/4751
result = result._unstack_full_reindex(dim, fill_value, sparse)
return result

def update(self, other: "CoercibleMapping") -> "Dataset":
Expand Down