Support __array_ufunc__ for xarray objects. #1962

shoyer · 2018-03-05T02:36:20Z

This means NumPy ufuncs are now supported directly on xarray.Dataset objects,
and opens the door to supporting computation on new data types, such as sparse
arrays or arrays with units.

Closes __array_ufunc__ for xarray #1617 (remove if there is no corresponding issue, which should only be the case for minor changes)
Tests added (for all bug fixes or enhancements)
Tests passed (for all non-documentation changes)
Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

This means NumPy ufuncs are now supported directly on xarray.Dataset objects, and opens the door to supporting computation on new data types, such as sparse arrays or arrays with units. Fixes GH1617

stickler-ci · 2018-03-05T02:37:04Z

xarray/core/common.py

+                           dask='allowed')
+
+
+class DataWithCoords(SupportsArithmetic, AttrAccessMixin):


W1641 Implementing eq without also implementing hash

stickler-ci · 2018-03-05T02:46:56Z

xarray/core/common.py

@@ -235,7 +239,65 @@ def get_squeeze_dims(xarray_obj, dim, axis=None):
    return dim


-class BaseDataObject(AttrAccessMixin):
+class SupportsArithmetic(object):


W1641 Implementing eq without also implementing hash

stickler-ci · 2018-03-05T02:49:24Z

xarray/core/common.py

@@ -235,7 +239,65 @@ def get_squeeze_dims(xarray_obj, dim, axis=None):
    return dim


-class BaseDataObject(AttrAccessMixin):
+class SupportsArithmetic(object):  # noqa: W1641


W1641 Implementing eq without also implementing hash

Why doesn't # noqa: W1641 disable this warning?

Because it's a pylint warning, not a flake8 warning. It's because py3k is enabled. We could turn that checker off

https://pylint.readthedocs.io/en/latest/user_guide/message-control.html

Yep, that's what I just did.

fujiisoup

This looks nice.
I could not find any edge cases that break backward compatibility.

shoyer · 2018-03-08T02:20:21Z

One potential edge case is if someone directly calls a ufunc reduce method, e.g., np.add.reduce. Previously, this would cast the xarray object to a numpy array, but now it will raise an error.

Example:

# current xarray
In [3]: np.add.reduce(xr.DataArray([1]))
Out[3]: 1

# with this pull request
In [3]: np.add.reduce(xr.DataArray(0))
NotImplementedError: reduce method for ufunc <ufunc 'add'> is not implemented on xarray objects, which currently only support the __call__ method.

Note that the more commonly used aliases for these reduce methods, e.g., np.sum() will continue to work since they check for sum() method on their argument.

There are also a few other ufunc methods that get used occasionally.

I think I'm OK breaking these because usage is so rare (and the work-around of casting to numpy arrays is so easy) but this should probably be noted in the release notes.

dopplershift · 2018-03-08T22:43:26Z

This looks awesome. Thoughts on where you think units fits in here?

shoyer · 2018-03-08T23:17:52Z

@dopplershift see #1938. Units should be able to make use of the same machinery as sparse arrays (__array_ufunc__ and multipledispatch), but xarray itself implementing __array_ufunc__ is not particularly useful, because you still can't put your own custom array types inside xarray objects.

dopplershift · 2018-03-09T01:13:16Z

At this point I'd be happy to have hooks that let me intercept/wrap ufunc operations, though I guess that's what #1938 is supporting in a more systematic way.

shoyer · 2018-03-09T01:16:42Z

If you try to put a pint array into xarray.DataArray right now, I'm pretty sure it will get cast into a NumPy array.

dopplershift · 2018-03-09T01:20:03Z

Right. But such hooks would be sufficient to properly maintain the units attribute on a DataArray and check whether math made sense. This could use pint under the covers.

jhamman

Looks great. Excited to get this working.

jhamman · 2018-03-09T04:29:20Z

xarray/core/common.py

 import warnings

 import numpy as np
 import pandas as pd

 from . import dtypes, formatting, ops
-from .pycompat import OrderedDict, basestring, dask_array_type, suppress
+from .arithmetic import SupportsArithmetic
+from .options import OPTIONS


is OPTIONS ever used or is it needed in this scope for some reason?

Oops we don't need it here.

Since commit c6ea042 (merged in atmtools#143), UnitsAwareDataArray depends on xarray.DataArray.__array_ufunc__ (which in turn depends on numpy 1.13 or newer). This was merged into xarray in pydata/xarray#1962 and added to release 0.10.2. The old UnitsAwareDataArray does not work with xarray>=0.10.2 and the new one does not work with xarray<=0.10.1. Therefore, typhon now depends on xarray>=0.10.2 and numpy>=1.13.

Support __array_ufunc__ for xarray objects.

a427b80

This means NumPy ufuncs are now supported directly on xarray.Dataset objects, and opens the door to supporting computation on new data types, such as sparse arrays or arrays with units. Fixes GH1617

stickler-ci reviewed Mar 5, 2018

View reviewed changes

shoyer added 3 commits March 4, 2018 18:40

add TODO note on xarray objects in out argument

661c5b4

Satisfy stickler for __eq__ overload

0ebbcb6

Move dummy arithmetic implementations to SupportsArithemtic

561ac77

stickler-ci reviewed Mar 5, 2018

View reviewed changes

Try again to disable flake8 warning

52c750d

stickler-ci reviewed Mar 5, 2018

View reviewed changes

shoyer added 2 commits March 4, 2018 19:47

Disable py3k tool on stickler-ci

0369786

Move arithmetic to its own file.

4e6ac28

fujiisoup approved these changes Mar 8, 2018

View reviewed changes

Merge branch 'master' into array-ufunc

1fc2844

jhamman approved these changes Mar 9, 2018

View reviewed changes

Remove unused imports

b6bed5b

shoyer mentioned this pull request Mar 9, 2018

0.10.2 release #1975

Closed

3 tasks

shoyer and others added 2 commits March 8, 2018 21:09

Add note on backwards incompatible changes from apply_ufunc

259a109

Merge branch 'master' into array-ufunc

8f4840e

shoyer merged commit b430524 into pydata:master Mar 12, 2018

gerritholl mentioned this pull request Apr 5, 2018

Add necessary version for xarray dependency atmtools/typhon#145

Merged

NotSqrt mentioned this pull request Jun 4, 2018

operation on complex number data #553

Closed

DocOtak mentioned this pull request Sep 3, 2020

Plans for full xarray compatibility? TEOS-10/GSW-Python#66

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support __array_ufunc__ for xarray objects. #1962

Support __array_ufunc__ for xarray objects. #1962

shoyer commented Mar 5, 2018

stickler-ci Mar 5, 2018

stickler-ci Mar 5, 2018

stickler-ci Mar 5, 2018

shoyer Mar 5, 2018

max-sixty Mar 5, 2018

shoyer Mar 5, 2018

fujiisoup left a comment

shoyer commented Mar 8, 2018

dopplershift commented Mar 8, 2018

shoyer commented Mar 8, 2018

dopplershift commented Mar 9, 2018

shoyer commented Mar 9, 2018

dopplershift commented Mar 9, 2018

jhamman left a comment

jhamman Mar 9, 2018

shoyer Mar 9, 2018

		dask='allowed')


		class DataWithCoords(SupportsArithmetic, AttrAccessMixin):

Support __array_ufunc__ for xarray objects. #1962

Support __array_ufunc__ for xarray objects. #1962

Conversation

shoyer commented Mar 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fujiisoup left a comment

Choose a reason for hiding this comment

shoyer commented Mar 8, 2018

dopplershift commented Mar 8, 2018

shoyer commented Mar 8, 2018

dopplershift commented Mar 9, 2018

shoyer commented Mar 9, 2018

dopplershift commented Mar 9, 2018

jhamman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment