Merge branch 'groupby-aggs-using-numpy-groupies' of github.com:anders…

…y005/xarray into groupby_npg

* 'groupby-aggs-using-numpy-groupies' of github.com:andersy005/xarray:
  Bump actions/github-script from 4.0.2 to 4.1 (pydata#5730)
  Set coord name concat when `concat`ing along a DataArray (pydata#5611)
  Add .git-blame-ignore-revs (pydata#5708)
  Type annotate tests (pydata#5728)
  Consolidate TypeVars in a single place (pydata#5569)
  add storage_options arg to to_zarr (pydata#5615)
  dataset `__repr__` updates (pydata#5580)
  Xfail failing test on main (pydata#5729)
  Add xarray-dataclasses to ecosystem in docs (pydata#5725)
  extend show_versions (pydata#5724)
  Move docstring for xr.set_options to numpy style (pydata#5702)
  Refactor more groupby and resample tests (pydata#5707)
  Remove suggestion to install pytest-xdist in docs (pydata#5713)
  Add typing to the OPTIONS dict (pydata#5678)
  Change annotations to allow str keys (pydata#5690)
  Whatsnew for float-to-top (pydata#5714)
  Use isort's float-to-top (pydata#5695)
  Fix errors in test_latex_name_isnt_split for min environments (pydata#5710)
  Improves rendering of complex LaTeX expressions as `long_name`s when plotting (pydata#5682)
  Use same bool validator as other inputs (pydata#5703)

.git-blame-ignore-revs

@@ -0,0 +1,5 @@
+# black PR 3142
+d089df385e737f71067309ff7abae15994d581ec
+
+# isort PR 1924
+0e73e240107caee3ffd1a1149f0150c390d43251

.github/workflows/upstream-dev-ci.yaml

@@ -122,7 +122,7 @@ jobs:
           shopt -s globstar
           python .github/workflows/parse_logs.py logs/**/*-log
       - name: Report failures
-        uses: actions/github-script@v4.0.2
+        uses: actions/github-script@v4.1
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |

doc/contributing.rst

@@ -628,13 +628,7 @@ Or with one of the following constructs::
     pytest xarray/tests/[test-module].py::[TestClass]::[test_method]
 
 Using `pytest-xdist <https://pypi.python.org/pypi/pytest-xdist>`_, one can
-speed up local testing on multicore machines. To use this feature, you will
-need to install `pytest-xdist` via::
-
-    pip install pytest-xdist
-
-
-Then, run pytest with the optional -n argument::
+speed up local testing on multicore machines, by running pytest with the optional -n argument::
 
     pytest xarray -n 4
 

doc/ecosystem.rst

@@ -68,7 +68,7 @@ Extend xarray capabilities
 - `hypothesis-gufunc <https://hypothesis-gufunc.readthedocs.io/en/latest/>`_: Extension to hypothesis. Makes it easy to write unit tests with xarray objects as input.
 - `nxarray <https://github.com/nxarray/nxarray>`_: NeXus input/output capability for xarray.
 - `xarray-compare <https://github.com/astropenguin/xarray-compare>`_: xarray extension for data comparison.
-- `xarray-custom <https://github.com/astropenguin/xarray-custom>`_: Data classes for custom xarray creation.
+- `xarray-dataclasses <https://github.com/astropenguin/xarray-dataclasses>`_: xarray extension for typed DataArray and Dataset creation.
 - `xarray_extras <https://github.com/crusaderky/xarray_extras>`_: Advanced algorithms for xarray objects (e.g. integrations/interpolations).
 - `xpublish <https://xpublish.readthedocs.io/>`_: Publish Xarray Datasets via a Zarr compatible REST API.
 - `xrft <https://github.com/rabernat/xrft>`_: Fourier transforms for xarray data.

doc/whats-new.rst

@@ -22,15 +22,25 @@ v0.19.1 (unreleased)
 
 New Features
 ~~~~~~~~~~~~
+- Xarray now does a better job rendering variable names that are long LaTeX sequences when plotting (:issue:`5681`, :pull:`5682`).
+  By `Tomas Chor <https://github.com/tomchor>`_.
 - Add a option to disable the use of ``bottleneck`` (:pull:`5560`)
   By `Justus Magin <https://github.com/keewis>`_.
 - Added ``**kwargs`` argument to :py:meth:`open_rasterio` to access overviews (:issue:`3269`).
   By `Pushkar Kopparla <https://github.com/pkopparla>`_.
+- Added ``storage_options`` argument to :py:meth:`to_zarr` (:issue:`5601`).
+  By `Ray Bell <https://github.com/raybellwaves>`_, `Zachary Blackwood <https://github.com/blackary>`_ and
+  `Nathan Lis <https://github.com/wxman22>`_.
 
 
 Breaking changes
 ~~~~~~~~~~~~~~~~
 
+- The ``__repr__`` of a :py:class:`xarray.Dataset`'s ``coords`` and ``data_vars``
+  ignore ``xarray.set_option(display_max_rows=...)`` and show the full output
+  when called directly as, e.g., ``ds.data_vars`` or ``print(ds.data_vars)``
+  (:issue:`5545`, :pull:`5580`).
+  By `Stefan Bender <https://github.com/st-bender>`_.
 
 Deprecations
 ~~~~~~~~~~~~
@@ -51,8 +61,15 @@ Internal Changes
   By `Deepak Cherian <https://github.com/dcherian>`_.
 - Explicit indexes refactor: decouple ``xarray.Index``` from ``xarray.Variable`` (:pull:`5636`).
   By `Benoit Bovy <https://github.com/benbovy>`_.
+- Fix ``Mapping`` argument typing to allow mypy to pass on ``str`` keys (:pull:`5690`).
+  By `Maximilian Roos <https://github.com/max-sixty>`_.
+- Annotate many of our tests, and fix some of the resulting typing errors. This will
+  also mean our typing annotations are tested as part of CI. (:pull:`5728`).
+  By `Maximilian Roos <https://github.com/max-sixty>`_.
 - Improve the performance of reprs for large datasets or dataarrays. (:pull:`5661`)
   By `Jimmy Westling <https://github.com/illviljan>`_.
+- Use isort's `float_to_top` config. (:pull:`5695`).
+  By `Maximilian Roos <https://github.com/max-sixty>`_.
 
 .. _whats-new.0.19.0:
 

properties/test_encode_decode.py

@@ -4,9 +4,10 @@
 These ones pass, just as you'd hope!
 
 """
-import pytest  # isort:skip
+import pytest
 
 pytest.importorskip("hypothesis")
+# isort: split
 
 import hypothesis.extra.numpy as npst
 import hypothesis.strategies as st
@@ -24,7 +25,7 @@
 
 @pytest.mark.slow
 @given(st.data(), an_array)
-def test_CFMask_coder_roundtrip(data, arr):
+def test_CFMask_coder_roundtrip(data, arr) -> None:
     names = data.draw(
         st.lists(st.text(), min_size=arr.ndim, max_size=arr.ndim, unique=True).map(
             tuple
@@ -38,7 +39,7 @@ def test_CFMask_coder_roundtrip(data, arr):
 
 @pytest.mark.slow
 @given(st.data(), an_array)
-def test_CFScaleOffset_coder_roundtrip(data, arr):
+def test_CFScaleOffset_coder_roundtrip(data, arr) -> None:
     names = data.draw(
         st.lists(st.text(), min_size=arr.ndim, max_size=arr.ndim, unique=True).map(
             tuple

properties/test_pandas_roundtrip.py

@@ -28,7 +28,7 @@
 
 
 @st.composite
-def datasets_1d_vars(draw):
+def datasets_1d_vars(draw) -> xr.Dataset:
     """Generate datasets with only 1D variables
 
     Suitable for converting to pandas dataframes.
@@ -49,7 +49,7 @@ def datasets_1d_vars(draw):
 
 
 @given(st.data(), an_array)
-def test_roundtrip_dataarray(data, arr):
+def test_roundtrip_dataarray(data, arr) -> None:
     names = data.draw(
         st.lists(st.text(), min_size=arr.ndim, max_size=arr.ndim, unique=True).map(
             tuple
@@ -62,15 +62,15 @@ def test_roundtrip_dataarray(data, arr):
 
 
 @given(datasets_1d_vars())
-def test_roundtrip_dataset(dataset):
+def test_roundtrip_dataset(dataset) -> None:
     df = dataset.to_dataframe()
     assert isinstance(df, pd.DataFrame)
     roundtripped = xr.Dataset(df)
     xr.testing.assert_identical(dataset, roundtripped)
 
 
 @given(numeric_series, st.text())
-def test_roundtrip_pandas_series(ser, ix_name):
+def test_roundtrip_pandas_series(ser, ix_name) -> None:
     # Need to name the index, otherwise Xarray calls it 'dim_0'.
     ser.index.name = ix_name
     arr = xr.DataArray(ser)
@@ -87,7 +87,7 @@ def test_roundtrip_pandas_series(ser, ix_name):
 
 @pytest.mark.xfail
 @given(numeric_homogeneous_dataframe)
-def test_roundtrip_pandas_dataframe(df):
+def test_roundtrip_pandas_dataframe(df) -> None:
     # Need to name the indexes, otherwise Xarray names them 'dim_0', 'dim_1'.
     df.index.name = "rows"
     df.columns.name = "cols"

setup.cfg

@@ -161,7 +161,7 @@ exclude=
 [isort]
 profile = black
 skip_gitignore = true
-force_to_top = true
+float_to_top = true
 default_section = THIRDPARTY
 known_first_party = xarray
 

xarray/backends/api.py

@@ -1319,6 +1319,7 @@ def to_zarr(
     append_dim: Hashable = None,
     region: Mapping[str, slice] = None,
     safe_chunks: bool = True,
+    storage_options: Dict[str, str] = None,
 ):
     """This function creates an appropriate datastore for writing a dataset to
     a zarr ztore
@@ -1330,6 +1331,22 @@ def to_zarr(
     store = _normalize_path(store)
     chunk_store = _normalize_path(chunk_store)
 
+    if storage_options is None:
+        mapper = store
+        chunk_mapper = chunk_store
+    else:
+        from fsspec import get_mapper
+
+        if not isinstance(store, str):
+            raise ValueError(
+                f"store must be a string to use storage_options. Got {type(store)}"
+            )
+        mapper = get_mapper(store, **storage_options)
+        if chunk_store is not None:
+            chunk_mapper = get_mapper(chunk_store, **storage_options)
+        else:
+            chunk_mapper = chunk_store
+
     if encoding is None:
         encoding = {}
 
@@ -1372,13 +1389,13 @@ def to_zarr(
         already_consolidated = False
         consolidate_on_close = consolidated or consolidated is None
     zstore = backends.ZarrStore.open_group(
-        store=store,
+        store=mapper,
         mode=mode,
         synchronizer=synchronizer,
         group=group,
         consolidated=already_consolidated,
         consolidate_on_close=consolidate_on_close,
-        chunk_store=chunk_store,
+        chunk_store=chunk_mapper,
         append_dim=append_dim,
         write_region=region,
         safe_chunks=safe_chunks,

xarray/backends/zarr.py

@@ -713,6 +713,9 @@ def open_zarr(
         falling back to read non-consolidated metadata if that fails.
     chunk_store : MutableMapping, optional
         A separate Zarr store only for chunk data.
+    storage_options : dict, optional
+        Any additional parameters for the storage backend (ignored for local
+        paths).
     decode_timedelta : bool, optional
         If True, decode variables and coordinates with time units in
         {'days', 'hours', 'minutes', 'seconds', 'milliseconds', 'microseconds'}

xarray/core/_typed_ops.pyi

@@ -9,24 +9,23 @@ from .dataarray import DataArray
 from .dataset import Dataset
 from .groupby import DataArrayGroupBy, DatasetGroupBy, GroupBy
 from .npcompat import ArrayLike
+from .types import (
+    DaCompatible,
+    DsCompatible,
+    GroupByIncompatible,
+    ScalarOrArray,
+    T_DataArray,
+    T_Dataset,
+    T_Variable,
+    VarCompatible,
+)
 from .variable import Variable
 
 try:
     from dask.array import Array as DaskArray
 except ImportError:
     DaskArray = np.ndarray
 
-# DatasetOpsMixin etc. are parent classes of Dataset etc.
-T_Dataset = TypeVar("T_Dataset", bound="DatasetOpsMixin")
-T_DataArray = TypeVar("T_DataArray", bound="DataArrayOpsMixin")
-T_Variable = TypeVar("T_Variable", bound="VariableOpsMixin")
-
-ScalarOrArray = Union[ArrayLike, np.generic, np.ndarray, DaskArray]
-DsCompatible = Union[Dataset, DataArray, Variable, GroupBy, ScalarOrArray]
-DaCompatible = Union[DataArray, Variable, DataArrayGroupBy, ScalarOrArray]
-VarCompatible = Union[Variable, ScalarOrArray]
-GroupByIncompatible = Union[Variable, GroupBy]
-
 class DatasetOpsMixin:
     __slots__ = ()
     def _binary_op(self, other, f, reflexive=...): ...

xarray/core/accessor_str.py

@@ -114,7 +114,7 @@ def _apply_str_ufunc(
     obj: Any,
     dtype: Union[str, np.dtype, Type] = None,
     output_core_dims: Union[list, tuple] = ((),),
-    output_sizes: Mapping[Hashable, int] = None,
+    output_sizes: Mapping[Any, int] = None,
     func_args: Tuple = (),
     func_kwargs: Mapping = {},
 ) -> Any:
@@ -227,7 +227,7 @@ def _apply(
         func: Callable,
         dtype: Union[str, np.dtype, Type] = None,
         output_core_dims: Union[list, tuple] = ((),),
-        output_sizes: Mapping[Hashable, int] = None,
+        output_sizes: Mapping[Any, int] = None,
         func_args: Tuple = (),
         func_kwargs: Mapping = {},
     ) -> Any:

xarray/core/common.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import warnings
 from contextlib import suppress
 from html import escape
@@ -36,10 +38,10 @@
 if TYPE_CHECKING:
     from .dataarray import DataArray
     from .dataset import Dataset
+    from .types import T_DataWithCoords, T_Xarray
     from .variable import Variable
     from .weighted import Weighted
 
-T_DataWithCoords = TypeVar("T_DataWithCoords", bound="DataWithCoords")
 
 C = TypeVar("C")
 T = TypeVar("T")
@@ -409,7 +411,7 @@ def get_index(self, key: Hashable) -> pd.Index:
             return pd.Index(range(self.sizes[key]), name=key)
 
     def _calc_assign_results(
-        self: C, kwargs: Mapping[Hashable, Union[T, Callable[[C], T]]]
+        self: C, kwargs: Mapping[Any, Union[T, Callable[[C], T]]]
     ) -> Dict[Hashable, T]:
         return {k: v(self) if callable(v) else v for k, v in kwargs.items()}
 
@@ -795,9 +797,7 @@ def groupby_bins(
             },
         )
 
-    def weighted(
-        self: T_DataWithCoords, weights: "DataArray"
-    ) -> "Weighted[T_DataWithCoords]":
+    def weighted(self: T_DataWithCoords, weights: "DataArray") -> Weighted[T_Xarray]:
         """
         Weighted operations.
 
@@ -818,9 +818,9 @@ def weighted(
 
     def rolling(
         self,
-        dim: Mapping[Hashable, int] = None,
+        dim: Mapping[Any, int] = None,
         min_periods: int = None,
-        center: Union[bool, Mapping[Hashable, bool]] = False,
+        center: Union[bool, Mapping[Any, bool]] = False,
         **window_kwargs: int,
     ):
         """
@@ -892,7 +892,7 @@ def rolling(
 
     def rolling_exp(
         self,
-        window: Mapping[Hashable, int] = None,
+        window: Mapping[Any, int] = None,
         window_type: str = "span",
         **window_kwargs,
     ):
@@ -933,9 +933,9 @@ def rolling_exp(
 
     def coarsen(
         self,
-        dim: Mapping[Hashable, int] = None,
+        dim: Mapping[Any, int] = None,
         boundary: str = "exact",
-        side: Union[str, Mapping[Hashable, str]] = "left",
+        side: Union[str, Mapping[Any, str]] = "left",
         coord_func: str = "mean",
         **window_kwargs: int,
     ):
@@ -1009,7 +1009,7 @@ def coarsen(
 
     def resample(
         self,
-        indexer: Mapping[Hashable, str] = None,
+        indexer: Mapping[Any, str] = None,
         skipna=None,
         closed: str = None,
         label: str = None,
@@ -1520,7 +1520,7 @@ def __getitem__(self, value):
 def full_like(
     other: "Dataset",
     fill_value,
-    dtype: Union[DTypeLike, Mapping[Hashable, DTypeLike]] = None,
+    dtype: Union[DTypeLike, Mapping[Any, DTypeLike]] = None,
 ) -> "Dataset":
     ...