Merge remote-tracking branch 'upstream/master' into fix/allow_lazy

* upstream/master: (27 commits)
  drop_vars; deprecate drop for variables (pydata#3475)
  uamiv test using only raw uamiv variables (pydata#3485)
  Optimize dask array equality checks. (pydata#3453)
  Propagate indexes in DataArray binary operations. (pydata#3481)
  python 3.8 tests (pydata#3477)
  __dask_tokenize__ (pydata#3446)
  Type check sentinel values (pydata#3472)
  Fix typo in docstring (pydata#3474)
  fix test suite warnings re `drop` (pydata#3460)
  Fix integrate docs (pydata#3469)
  Fix leap year condition in monthly means example (pydata#3464)
  Hypothesis tests for roundtrip to & from pandas (pydata#3285)
  unpin cftime (pydata#3463)
  Cleanup whatsnew (pydata#3462)
  enable xr.ALL_DIMS in xr.dot (pydata#3424)
  Merge stable into master (pydata#3457)
  upgrade black verison to 19.10b0 (pydata#3456)
  Remove outdated code related to compatibility with netcdftime (pydata#3450)
  Remove deprecated behavior from dataset.drop docstring (pydata#3451)
  jupyterlab dark theme (pydata#3443)
  ...

MANIFEST.in

@@ -6,3 +6,4 @@ prune doc/generated
 global-exclude .DS_Store
 include versioneer.py
 include xarray/_version.py
+recursive-include xarray/static *

README.rst

@@ -138,4 +138,7 @@ under a "3-clause BSD" license:
 xarray also bundles portions of CPython, which is available under the "Python
 Software Foundation License" in xarray/core/pycompat.py.
 
+xarray uses icons from the icomoon package (free version), which is
+available under the "CC BY 4.0" license.
+
 The full text of these licenses are included in the licenses directory.

azure-pipelines.yml

@@ -18,6 +18,8 @@ jobs:
         conda_env: py36
       py37:
         conda_env: py37
+      py38:
+        conda_env: py38
       py37-upstream-dev:
         conda_env: py37
         upstream_dev: true

ci/requirements/py36-min-all-deps.yml

@@ -13,7 +13,7 @@ dependencies:
   - cartopy=0.17
   - cdms2=3.1
   - cfgrib=0.9
-  - cftime=1.0.3  # FIXME need 1.0.5 (not released yet); 1.0.4 is broken
+  - cftime=1.0
   - coveralls
   - dask=1.2
   - distributed=1.27

ci/requirements/py36.yml

@@ -9,7 +9,7 @@ dependencies:
   - cartopy
   - cdms2
   - cfgrib
-  - cftime<1.0.4  # FIXME need 1.0.5 (not released yet); 1.0.4 is broken
+  - cftime
   - coveralls
   - dask
   - distributed

ci/requirements/py37-windows.yml

@@ -9,7 +9,7 @@ dependencies:
   - cartopy
   # - cdms2  # Not available on Windows
   # - cfgrib  # Causes Python interpreter crash on Windows
-  - cftime<1.0.4  # FIXME need 1.0.5 (not released yet); 1.0.4 is broken
+  - cftime
   - coveralls
   - dask
   - distributed

ci/requirements/py37.yml

@@ -9,7 +9,7 @@ dependencies:
   - cartopy
   - cdms2
   - cfgrib
-  - cftime<1.0.4  # FIXME need 1.0.5 (not released yet); 1.0.4 is broken
+  - cftime
   - coveralls
   - dask
   - distributed

ci/requirements/py38.yml

@@ -0,0 +1,15 @@
+name: xarray-tests
+channels:
+  - conda-forge
+dependencies:
+  - python=3.8
+  - pip
+  - pip:
+      - coveralls
+      - dask
+      - distributed
+      - numpy
+      - pandas
+      - pytest
+      - pytest-cov
+      - pytest-env

doc/data-structures.rst

@@ -393,14 +393,14 @@ methods (like pandas) for transforming datasets into new objects.
 
 For removing variables, you can select and drop an explicit list of
 variables by indexing with a list of names or using the
-:py:meth:`~xarray.Dataset.drop` methods to return a new ``Dataset``. These
+:py:meth:`~xarray.Dataset.drop_vars` methods to return a new ``Dataset``. These
 operations keep around coordinates:
 
 .. ipython:: python
 
     ds[['temperature']]
     ds[['temperature', 'temperature_double']]
-    ds.drop('temperature')
+    ds.drop_vars('temperature')
 
 To remove a dimension, you can use :py:meth:`~xarray.Dataset.drop_dims` method.
 Any variables using that dimension are dropped:
@@ -411,7 +411,7 @@ Any variables using that dimension are dropped:
 
 As an alternate to dictionary-like modifications, you can use
 :py:meth:`~xarray.Dataset.assign` and :py:meth:`~xarray.Dataset.assign_coords`.
-These methods return a new dataset with additional (or replaced) or values:
+These methods return a new dataset with additional (or replaced) values:
 
 .. ipython:: python
 
@@ -420,7 +420,7 @@ These methods return a new dataset with additional (or replaced) or values:
 There is also the :py:meth:`~xarray.Dataset.pipe` method that allows you to use
 a method call with an external function (e.g., ``ds.pipe(func)``) instead of
 simply calling it (e.g., ``func(ds)``). This allows you to write pipelines for
-transforming you data (using "method chaining") instead of writing hard to
+transforming your data (using "method chaining") instead of writing hard to
 follow nested function calls:
 
 .. ipython:: python

doc/examples/monthly-means.rst

@@ -83,7 +83,7 @@ the ``calendar.month_range`` function.
 
         for i, (month, year) in enumerate(zip(time.month, time.year)):
             month_length[i] = cal_days[month]
-            if leap_year(year, calendar=calendar):
+            if leap_year(year, calendar=calendar) and month == 2:
                 month_length[i] += 1
         return month_length
 

doc/examples/multidimensional-coords.rst

@@ -107,7 +107,7 @@ function to specify the output coordinates of the group.
     lat_center = np.arange(1, 90, 2)
     # group according to those bins and take the mean
     Tair_lat_mean = (ds.Tair.groupby_bins('xc', lat_bins, labels=lat_center)
-	             .mean(xr.ALL_DIMS))
+	             .mean(...))
     # plot the result
     @savefig xarray_multidimensional_coords_14_1.png width=5in
     Tair_lat_mean.plot();

doc/groupby.rst

@@ -116,7 +116,13 @@ dimensions *other than* the provided one:
 
 .. ipython:: python
 
-    ds.groupby('x').std(xr.ALL_DIMS)
+    ds.groupby('x').std(...)
+
+.. note::
+
+    We use an ellipsis (`...`) here to indicate we want to reduce over all
+    other dimensions  
+
 
 First and last
 ~~~~~~~~~~~~~~
@@ -127,7 +133,7 @@ values for group along the grouped dimension:
 
 .. ipython:: python
 
-    ds.groupby('letters').first(xr.ALL_DIMS)
+    ds.groupby('letters').first(...)
 
 By default, they skip missing values (control this with ``skipna``).
 
@@ -142,7 +148,7 @@ coordinates. For example:
 
 .. ipython:: python
 
-    alt = arr.groupby('letters').mean(xr.ALL_DIMS)
+    alt = arr.groupby('letters').mean(...)
     alt
     ds.groupby('letters') - alt
 
@@ -195,7 +201,7 @@ __ http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_two_dimen
                 'lat': (['ny','nx'], [[10,10],[20,20]] ),},
         dims=['ny','nx'])
     da
-    da.groupby('lon').sum(xr.ALL_DIMS)
+    da.groupby('lon').sum(...)
     da.groupby('lon').apply(lambda x: x - x.mean(), shortcut=False)
 
 Because multidimensional groups have the ability to generate a very large
@@ -213,4 +219,4 @@ applying your function, and then unstacking the result:
 .. ipython:: python
 
    stacked = da.stack(gridcell=['ny', 'nx'])
-   stacked.groupby('gridcell').sum(xr.ALL_DIMS).unstack('gridcell')
+   stacked.groupby('gridcell').sum(...).unstack('gridcell')

doc/indexing.rst

@@ -232,14 +232,14 @@ Using indexing to *assign* values to a subset of dataset (e.g.,
 Dropping labels and dimensions
 ------------------------------
 
-The :py:meth:`~xarray.Dataset.drop` method returns a new object with the listed
+The :py:meth:`~xarray.Dataset.drop_sel` method returns a new object with the listed
 index labels along a dimension dropped:
 
 .. ipython:: python
 
-    ds.drop(space=['IN', 'IL'])
+    ds.drop_sel(space=['IN', 'IL'])
 
-``drop`` is both a ``Dataset`` and ``DataArray`` method.
+``drop_sel`` is both a ``Dataset`` and ``DataArray`` method.
 
 Use :py:meth:`~xarray.Dataset.drop_dims` to drop a full dimension from a Dataset.
 Any variables with these dimensions are also dropped:

doc/reshaping.rst

@@ -18,12 +18,14 @@ Reordering dimensions
 ---------------------
 
 To reorder dimensions on a :py:class:`~xarray.DataArray` or across all variables
-on a :py:class:`~xarray.Dataset`, use :py:meth:`~xarray.DataArray.transpose`:
+on a :py:class:`~xarray.Dataset`, use :py:meth:`~xarray.DataArray.transpose`. An 
+ellipsis (`...`) can be use to represent all other dimensions:
 
 .. ipython:: python
 
     ds = xr.Dataset({'foo': (('x', 'y', 'z'), [[[42]]]), 'bar': (('y', 'z'), [[24]])})
     ds.transpose('y', 'z', 'x')
+    ds.transpose(..., 'x')  # equivalent
     ds.transpose()  # reverses all dimensions
 
 Expand and squeeze dimensions

doc/terminology.rst

@@ -15,7 +15,7 @@ Terminology
 
 ----
 
-**Variable:** A `NetCDF-like variable <https://www.unidata.ucar.edu/software/netcdf/netcdf/Variables.html>`_ consisting of dimensions, data, and attributes which describe a single array. The main functional difference between variables and numpy arrays is that numerical operations on variables implement array broadcasting by dimension name. Each ``DataArray`` has an underlying variable that can be accessed via ``arr.variable``. However, a variable is not fully described outside of either a ``Dataset`` or a ``DataArray``.
+**Variable:** A `NetCDF-like variable <https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html#variables>`_ consisting of dimensions, data, and attributes which describe a single array. The main functional difference between variables and numpy arrays is that numerical operations on variables implement array broadcasting by dimension name. Each ``DataArray`` has an underlying variable that can be accessed via ``arr.variable``. However, a variable is not fully described outside of either a ``Dataset`` or a ``DataArray``.
 
 .. note::
 
@@ -39,4 +39,4 @@ Terminology
 
 ----
 
-**Index:** An *index* is a data structure optimized for efficient selecting and slicing of an associated array. Xarray creates indexes for dimension coordinates so that operations along dimensions are fast, while non-dimension coordinates are not indexed. Under the hood, indexes are implemented as :py:class:`pandas.Index` objects. The index associated with dimension name ``x`` can be retrieved by ``arr.indexes[x]``. By construction, ``len(arr.dims) == len(arr.indexes)``
+**Index:** An *index* is a data structure optimized for efficient selecting and slicing of an associated array. Xarray creates indexes for dimension coordinates so that operations along dimensions are fast, while non-dimension coordinates are not indexed. Under the hood, indexes are implemented as :py:class:`pandas.Index` objects. The index associated with dimension name ``x`` can be retrieved by ``arr.indexes[x]``. By construction, ``len(arr.dims) == len(arr.indexes)``

doc/whats-new.rst

@@ -21,49 +21,110 @@ v0.14.1 (unreleased)
 Breaking changes
 ~~~~~~~~~~~~~~~~
 
-- Minimum cftime version is now 1.0.3. By `Deepak Cherian <https://github.com/dcherian>`_.
-
-New Features
-~~~~~~~~~~~~
-- Added integration tests against `pint <https://pint.readthedocs.io/>`_.
-  (:pull:`3238`) by `Justus Magin <https://github.com/keewis>`_.
+- Broken compatibility with cftime < 1.0.3.
+  By `Deepak Cherian <https://github.com/dcherian>`_.
 
   .. note::
 
-    At the moment of writing, these tests *as well as the ability to use pint in general*
-    require `a highly experimental version of pint
-    <https://github.com/andrewgsavage/pint/pull/6>`_ (install with
-    ``pip install git+https://github.com/andrewgsavage/pint.git@refs/pull/6/head)``.
-    Even with it, interaction with non-numpy array libraries, e.g. dask or sparse, is broken.
+    cftime version 1.0.4 is broken
+    (`cftime/126 <https://github.com/Unidata/cftime/issues/126>`_);
+    please use version 1.0.4.2 instead.
+
+- All leftover support for dates from non-standard calendars through netcdftime, the
+  module included in versions of netCDF4 prior to 1.4 that eventually became the
+  cftime package, has been removed in favor of relying solely on the standalone
+  cftime package (:pull:`3450`).
+  By `Spencer Clark <https://github.com/spencerkclark>`_.
+
+New Features
+~~~~~~~~~~~~
+- :py:meth:`Dataset.drop_sel` & :py:meth:`DataArray.drop_sel` have been added for dropping labels.
+  :py:meth:`Dataset.drop_vars` & :py:meth:`DataArray.drop_vars` have been added for 
+  dropping variables (including coordinates). The existing ``drop`` methods remain as a backward compatible 
+  option for dropping either lables or variables, but using the more specific methods is encouraged.
+  (:pull:`3475`)
+  By `Maximilian Roos <https://github.com/max-sixty>`_
+- :py:meth:`Dataset.transpose` and :py:meth:`DataArray.transpose` now support an ellipsis (`...`)
+  to represent all 'other' dimensions. For example, to move one dimension to the front,
+  use `.transpose('x', ...)`. (:pull:`3421`)
+  By `Maximilian Roos <https://github.com/max-sixty>`_
+- Changed `xr.ALL_DIMS` to equal python's `Ellipsis` (`...`), and changed internal usages to use
+  `...` directly. As before, you can use this to instruct a `groupby` operation
+  to reduce over all dimensions. While we have no plans to remove `xr.ALL_DIMS`, we suggest
+  using `...`. (:pull:`3418`)
+  By `Maximilian Roos <https://github.com/max-sixty>`_
+- :py:func:`~xarray.dot`, and :py:func:`~xarray.DataArray.dot` now support the
+  `dims=...` option to sum over the union of dimensions of all input arrays
+  (:issue:`3423`) by `Mathias Hauser <https://github.com/mathause>`_.
+- Added new :py:meth:`Dataset._repr_html_` and :py:meth:`DataArray._repr_html_` to improve
+  representation of objects in jupyter. By default this feature is turned off
+  for now. Enable it with :py:meth:`xarray.set_options(display_style="html")`.
+  (:pull:`3425`) by `Benoit Bovy <https://github.com/benbovy>`_ and
+  `Julia Signell <https://github.com/jsignell>`_.
+- Implement `dask deterministic hashing
+  <https://docs.dask.org/en/latest/custom-collections.html#deterministic-hashing>`_
+  for xarray objects. Note that xarray objects with a dask.array backend already used
+  deterministic hashing in previous releases; this change implements it when whole
+  xarray objects are embedded in a dask graph, e.g. when :meth:`DataArray.map` is
+  invoked. (:issue:`3378`, :pull:`3446`)
+  By `Deepak Cherian <https://github.com/dcherian>`_ and
+  `Guido Imperiale <https://github.com/crusaderky>`_.
 
 Bug fixes
 ~~~~~~~~~
 - Fix regression introduced in v0.14.0 that would cause a crash if dask is installed
   but cloudpickle isn't (:issue:`3401`) by `Rhys Doyle <https://github.com/rdoyle45>`_
-
-- Sync with cftime by removing `dayofwk=-1` for cftime>=1.0.4. 
+- Fix grouping over variables with NaNs. (:issue:`2383`, :pull:`3406`).
+  By `Deepak Cherian <https://github.com/dcherian>`_.
+- Use dask names to compare dask objects prior to comparing values after computation.
+  (:issue:`3068`, :issue:`3311`, :issue:`3454`, :pull:`3453`).
+  By `Deepak Cherian <https://github.com/dcherian>`_.
+- Sync with cftime by removing `dayofwk=-1` for cftime>=1.0.4.
   By `Anderson Banihirwe <https://github.com/andersy005>`_.
 
 - Rolling reduction operations now longer compute dask arrays by default. (:issue:`3161`).
   In addition, the ``allow_lazy`` kwarg to ``reduce`` is deprecated.
   By `Deepak Cherian <https://github.com/dcherian>`_.
+- Fix :py:meth:`xarray.core.groupby.DataArrayGroupBy.reduce` and
+  :py:meth:`xarray.core.groupby.DatasetGroupBy.reduce` when reducing over multiple dimensions.
+  (:issue:`3402`). By `Deepak Cherian <https://github.com/dcherian/>`_
 
 Documentation
 ~~~~~~~~~~~~~
-
+- Fix leap year condition in example (http://xarray.pydata.org/en/stable/examples/monthly-means.html)
+  by `Mickaël Lalande <https://github.com/mickaellalande>`_.
 - Fix the documentation of :py:meth:`DataArray.resample` and
   :py:meth:`Dataset.resample` and explicitly state that a
   datetime-like dimension is required. (:pull:`3400`)
   By `Justus Magin <https://github.com/keewis>`_.
 - Update the terminology page to address multidimensional coordinates. (:pull:`3410`)
   By `Jon Thielen <https://github.com/jthielen>`_.
+- Fix the documentation of :py:meth:`Dataset.integrate` and
+  :py:meth:`DataArray.integrate` and add an example to
+  :py:meth:`Dataset.integrate`. (:pull:`3469`)
+  By `Justus Magin <https://github.com/keewis>`_.
 
 Internal Changes
 ~~~~~~~~~~~~~~~~
 
-- Use Python 3.6 idioms throughout the codebase. (:pull:3419)
+- Added integration tests against `pint <https://pint.readthedocs.io/>`_.
+  (:pull:`3238`) by `Justus Magin <https://github.com/keewis>`_.
+
+  .. note::
+
+    At the moment of writing, these tests *as well as the ability to use pint in general*
+    require `a highly experimental version of pint
+    <https://github.com/andrewgsavage/pint/pull/6>`_ (install with
+    ``pip install git+https://github.com/andrewgsavage/pint.git@refs/pull/6/head)``.
+    Even with it, interaction with non-numpy array libraries, e.g. dask or sparse, is broken.
+
+- Use Python 3.6 idioms throughout the codebase. (:pull:`3419`)
+  By `Maximilian Roos <https://github.com/max-sixty>`_
+
+- Run basic CI tests on Python 3.8. (:pull:`3477`)
   By `Maximilian Roos <https://github.com/max-sixty>`_
 
+
 .. _whats-new.0.14.0:
 
 v0.14.0 (14 Oct 2019)
@@ -3701,6 +3762,7 @@ Enhancements
   explicitly listed variables or index labels:
 
   .. ipython:: python
+     :okwarning:
 
       # drop variables
       ds = xray.Dataset({'x': 0, 'y': 1})