Insuring interoperability of pint with other (non-unit) array-like types

[jthielen]

When working with the new __array_function__ implementation in #764, seeing what it takes to get pint Quantity's working inside xarray (pydata/xarray#525), and trying to wrap my head around NEP 18, I realized that (to the best of what I could find) pint's expected behavior with other ndarray-like types is not well-documented or tested for. And so, I hope that this issue could be a discussion on clarifying those expected behaviors and determining how to document and test them.

To start that discussion, I think it would make sense to split other array-like types into two categories, those that should wrap pint, and those that should be wrapped by pint, or in other words, determine where pint should fall in the "type casting hierarchy". Based on pydata/xarray#525 (comment) and other comments I've seen, a split like the following seems to be desired:

Wrap pint Quantity

• xarray DataArray

Wrapped by pint Quantity

• dask array
• cupy array
• sparse array
• numpy masked array (and of course ndarray)

(Pandas seems to have special handling through pint-pandas, and hopefully others that I'm missing can be added to either list as appropriate)

Does this make sense as a division, and should a list like this be documented anywhere?

Also, once #764 is in place, it seems like there will be a need for explicit identification of types in each category...types that should wrap pint are used in

pint/pint/quantity.py

Line 70 in 4dcbe78

if other.__class__.__name__ in ["PintArray", "Series"]:

, and types that should be wrapped by pint would be used with implementing the functionality mentioned in #764. However, what should be done about array-like types that are unknown to pint, and how can consistent behavior between different Quantity-related operations be assured?

Finally, should pint implement tests for all the types it expects to be able to wrap? I got the impression from pydata/xarray#525 and pydata/xarray#2956 that tests fit best with the type higher in the type casting hierarchy (the one doing the wrapping).

Please do correct me on anything I am mistaken about with the above rambling 😄.

xref pydata/xarray#525, pydata/xarray#2956, #764, #633, Unidata/MetPy#996

EDIT: Discussion about arbitrarily nested metadata with NEP 18 implementers is ongoing at dask/dask#5329, which will be good to keep in mind here too.

[shoyer]

From a coordination perspective, I think the main task is to ensure that there's a well defined type casting hierarchy among all NEP-18 duck arrays, so it's clear which project is responsible for handling which interaction. This will help avoid incompatible nestings, e.g., dask wrapping pint and pint wrapping dask.

This might be something that makes sense as a continuously updated "Informational NEP" defining a defacto community standard.

Here's a partial attempt, with most of the duck-arrays I can think of off hand. The arrow X -> Y means that X handles interactions between X and Y arrays:

The general principle here is that if library X can wrap Y arrays, then it should also handle their interaction. This keeps the logic about how different array library interact in one place, as much as possible.

Black lines indicate existing relationships; gray lines indicate proposed relationships.

[hgrecco]

looks good to me

[shoyer]

Now generated with graphviz, which should be a little more maintainable:

[crusaderky]

@shoyer I changed the style of the arrows to dotted where there is only planned or untested support, and added arrows pint -> sparse and pint-> cupy

https://gist.github.com/crusaderky/1b00fd68ae1ce79109b6af546c88bd4a

[pentschev]

Those are nice graphs @shoyer, thanks for doing that.

The general principle here is that if library X can wrap Y arrays, then it should also handle their interaction. This keeps the logic about how different array library interact in one place, as much as possible.

I'm wondering if this would be a guideline that would be worth to add to the NEP-18? I feel it's quite a good idea to make it more visible, even if elsewhere.

[jthielen]

Some progress towards this goal (at least within Pint) has been made with #905, however, I think the following are the high-level tasks remaining:

• Type checks on Quantity creation to ensure type casting hierarchy is maintained (see Preserve ndarray-like magnitudes or reject them from the start #479)
• Rethink force_ndarray array behavior in _to_magnitude and elsewhere to handle casting to ndarray and casting to duck arrays (NEP-30 may be relevant here)
  • Implement force_ndarray_like as discussed here
  • Clearly document differences between force_ndarray, force_ndarray_like, and default
  • Until NEP 30 is implemented, may need to have an alternative heuristic for duck array (perhaps __array_function__ + shape/ndim/dtype?)
• Determine if the type casting hierarchy assumption used in NEP-18 Compatibility #905 is valid (Pint has a limited, pre-defined list of types that are above it in the hierarchy, and it will wrap all other array-like types)
  • Should the list of upcast types be made extensible? List is in public API at pint.compat.upcast_types
• Resolve Proper handling of __array_*__ attributes/methods #924
• Add tests with upcast types to ensure Pint defers (and commutes) properly
  • Pandas (deferred)
  • xarray
• Add tests with downcast types to ensure Pint wraps (and commutes with) them properly
  • Sparse
  • Dask (simple wrapping, further integration can be left to Integration with Dask (add tests; implement the Dask collection interface on Quantity) #883) (deferred)
  • CuPy (is special CI needed for GPU tests? In any case, this will be deferred)
  • Masked arrays (see Operating elementwise with numpy masked arrays #37, Inconsistency when multiplying NumPy masked arrays #633, masked array: The unit of the quantity is stripped. #688)
• Add documentation on scalar and array type compatibility (as discussed here and in NEP18 trouble when pint is being wrapped #878 and NEP-18 and python scalars #950)

[hgrecco]

Thanks @jthielen for summarizing this. I think is important to have input from people using heavily those packages to make it not only correct but also that it feels right. We can start by collecting some tests.

Another question that I have: should we have related packages (like pint-pandas) as a way to speed up development?

Some progress towards this goal (at least within Pint) has been made with #905,

A large progress has been made in #905, don't underestimate your work! 😄

[jthielen]

In light of #955, I hope to be working on most of the areas listed above in the next couple days. However, the tests with downcast types are likely to not all be ready for 0.10 because

• Dask is held up due to Type checking or duck typing inside dask.array.Array.__array_function__ dask/dask#4583
• I don't want to worry about GPU tests to test CuPy

However, I think testing Sparse and masked arrays as downcast types should be sufficient for now.

Also, for simplicity, I think I'll just do the upcast type tests with xarray, and not worry about Pandas for now.