⚠️ Nightly upstream-dev CI failed ⚠️

[github-actions]

Workflow Run URL

Python 3.10 Test Summary

tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tas-98-tg90p-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmin-98-tn90p-A-JUL-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmax-98-tx90p-Q-APR-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmax-98-tx90p-Q-APR-True-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmin-2-tn10p-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmax-2-tx10p-YS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmax-2-tx10p-YS-True-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tas-2-tg10p-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmax-98-warm_spell_duration_index-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[tasmin-2-cold_spell_duration_index-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[pr-99-days_over_precip_thresh-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[pr-98-fraction_over_precip_thresh-MS-False-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_bootstrapping.py::Test_bootstrap::test_bootstrap[pr-98-fraction_over_precip_thresh-MS-True-True]: ValueError: unmatched keys found in indexes and variables: {'year', 'dayofyear'}
tests/test_ensembles.py::TestEnsembleReduction::test_kmeans_rsqopt: assert [4, 5, 7, 10, 12, 13] == [3, 4, 5, 7, 10, 11, ...]
  At index 0 diff: 4 != 3
  Right contains 2 more items, first extra item: 12
  Full diff:
  - [3, 4, 5, 7, 10, 11, 12, 13]
  ?  ---             ----
  + [4, 5, 7, 10, 12, 13]
tests/test_ensembles.py::TestEnsembleReduction::test_kmeans_sampleweights: assert [4, 5, 7, 10, 12, 13] == [4, 5, 7, 10, 11, 12, ...]
  At index 4 diff: 12 != 11
  Right contains one more item: 13
  Full diff:
  - [4, 5, 7, 10, 11, 12, 13]
  ?               ----
  + [4, 5, 7, 10, 12, 13]

[bzah]

I will try to have a look but I've been out of the loop for a few months, I might need help.

[Zeitsperre]

@bzah All good. We assigned you since the changes are affecting the bootstrap functions, but let us know if you need a hand / are unavailable. Thanks!

[Zeitsperre]

Reopening. #1418 was dealing with a previous nightly failure.

[bzah]

I'm not 100% sure yet, but it seems this PR on xarray broke our tests.
The PR seems to change the behavior of assign_coords and create a new abstraction class Coordinate in xarray.
In particular it breaks on percentile_doy at

rrr = rr.assign_coords(time=ind).unstack("time").stack(stack_dim=("year", "window"))

What is strange is that it's only when dask is used in the tests that it breaks (see the [True] at the end of the tests failure).
I still have to figure out why exactly it behaves like this.

[bzah]

I just tried to uninstall the xarray 2023.7.1.dev in favor of xarray 2023.7.0 (latest release) and the tests pass.
I don't have much time right now but I will try to investigate further next week.

[bzah]

I have a MRE but I still don't understand what's going on:

    import pandas as pd
    import xarray as xr
    import numpy as np
    
    climate_var = xr.DataArray(
        np.zeros(1461),
        coords=[pd.date_range("2000-01-01", periods=1461, freq="D")],
        dims="time",
        name="tas",
    )
    climate_var = climate_var.chunk(dict(time=50))
    rr = climate_var.rolling(min_periods=1, center=True, time=5).construct("window")
    ind = pd.MultiIndex.from_arrays(
        (rr.time.dt.year.values, rr.time.dt.dayofyear.values),
        names=("year", "dayofyear"),
    )
    rr.assign_coords(time=ind).unstack("time")

If the dataarray is not chunked everything works, if it is it raises:

ValueError: unmatched keys found in indexes and variables: {'dayofyear', 'year'}

Should we escalate this to xarray or rather dask ?

[aulemahal]

I can throw an eye on this today!

[aulemahal]

After the rr.assign_coords(time=ind), the DataArray prints as:

<xarray.DataArray 'tas' (time: 1461, window: 5)>
dask.array<sliding_window_view, shape=(1461, 5), dtype=float64, chunksize=(52, 5), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) object MultiIndex
Dimensions without coordinates: window

This is strange, I'm pretty sure year and dayofyear should be listed in the coordinates.

The investigation continues. 🕵️

[aulemahal]

Escalated to xarray: pydata/xarray#8039

[bzah]

Thanks for the fix Pascal !