v0.10.1 Release

[jhamman]

We're close to a minor/bug-fix release (0.10.1). What do we need to get done before that can happen?

• WIP: Performance improvements for zarr backend #1800 Performance improvements to Zarr (@jhamman)
• fix distributed writes #1793 Fix for to_netcdf writes with dask-distributed (@jhamman, could use help)
• Normalisation for RGB imshow #1819 Normalisation for RGB imshow

Help wanted / bugs that no-one is working on:

• Comparison with masked array yields object-array with nans for masked values #1792 Comparison to masked numpy arrays
• .groupby_bins fails when data is not contained in bins #1764 groupby_bins fails for empty bins

What else?

[dcherian]

https://github.com/pydata/xarray/issues?q=is%3Aissue+milestone%3A0.10.1+is%3Aopen

[ghost]

#1865
It's a simple fix, but it affects my ability to produce CF1.7 netCDF files for distribution of our satellite data. The pull request #1869 is ready to go.

[Zac-HD]

@jhamman & @shoyer - I think Xarray has a release frequency problem.

• Plot nans #1782 has been merged, but I still can't plot timeseries with an all-nan step
• Support RGB[A] arrays in plot.imshow() #1796, Normalisation for RGB imshow #1819, and Use correct dtype for RGB image alpha channel #1893 have been merged - but I still can't plot RGB images
• Read small integers as float32, not float64 #1840 has been merged - but loading CF-int-encoded data still uses double the memory it needs.
• All of these daily tasks for myself and my colleagues are harder than they should be.

It's been more than three months now without a patch release. This is really, really frustrating as an Xarray contributor, user, and advocate - getting my work merged upstream literally isn't worth anything until it's released, my colleagues have trouble using it (and go back to Matlab or IDL!), and it's harder to ask for anything in meetings with eg @opendatacube.

Moving to weekly patch releases would fix all of these problems.

Maintainer availability doesn't need to be a limiting factor, either - for example, @HypothesisWorks has a deployment pipeline where the only human involvement is to click 'merge', and I'd be happy to help out if you'd like to set up a similar system.

[shoyer]

@Zac-HD Thanks for raising these concerns! I agree there are only upsides to a faster release cycle, and three months is way too long given the long back-log of improvements we have. Even without more automation, we should not wait more than 2-4 weeks between releases unless there are breaking changes in process (which can sometimes require more refinement to get right).

I think the trouble is that it's easy to get in a bad pattern of waiting for "just one more pull request" before issuing a new release. This is self-reinforcing -- the fewer releases we have, the more important it is to wait for a pull request before issuing a release.

I'd love to explore a more automated release pipeline. Take a look here for our current release instructions: https://github.com/pydata/xarray/blob/master/HOW_TO_RELEASE

In my experience, the slowest step is clean-ups/fixes of the "what's new" docs page (e.g., what I just did in #1940). It's hard to do that as part of the regular PR process in part because we only see built docs after PRs are merged. That said, if we do this more regularly the work required decreases (I spent about half an hour on this today). Unfortunately, there is no automated substitute for a human looking over a document to verify that it looks right :).

The actual release issuing process is quite fast once you know what you're doing, taking as little as half an hour minutes to update pypi, ReadTheDocs and conda-forge. But it would certainly be great if anyone (even non-maintainers) is capable of triggering this with a simple PR. Ideally, we should be able to issue a release anytime someone is willing to put in the effort of looking things over and ensuring the docs look right.

Anyways, my plan is to issue a release later today or tomorrow, as soon as #1940 is merged. There are lots of other excellent improvements in the pipeline (rolling window improvements, distributed writes to netCDF files, etc.) but we should just issue another release when those get merged instead of holding things up!

[jhamman]

I'm a enthusiastic 👍 for getting this release out ASAP and generally releasing more regularly. I take some of the blame for moving the goalposts for this release, particularly with regard to #1464.

@shoyer - please go forward with the release and let me know if you need a hand with anything.

[shoyer]

I just pushed the release to pypi.

I'm dealing with the RTD build now, which seems to have broken without us noticing!

[jhamman]

@shoyer - I can do the conda-forge update.

[shoyer]

I'm going to push another version of the 0.10.1 tag to GitHub that includes fixes to doc/environment.yml (20207fd and f530e66).

This will allow us to actually build docs for 0.10.1 on ReadTheDocs.

[shoyer]

My messing around with tags seems to have been successful. The docs successfully build and been updated: http://xarray.pydata.org/en/stable/whats-new.html

It looks like the conda-forge update is also in progress: conda-forge/xarray-feedstock#27 (thanks @jhamman and @ocefpaf !)

[rabernat]

@Zac-HD: I hear your frustration! Thanks for putting some much needed pressure on this community to move to a faster release cycle. I think xarray is experiencing some growing pains as adoption increases. I generally favor automating as much as possible, modulo the necessary manual checks described by @shoyer.

[Zac-HD]

First: thanks, everyone, for such a prompt and helpful response! I'm excited both to have 10.1 (:tada:), and by the prospect of faster/automated releases in future.

Reading over the releasing instructions, I think there are three parts we need to work on to go fully automated. By fully automated, I mean "no maintainer action whatsoever beyond merging pulls, which are not release-specific":

• Most things can be automated with continuous deployment scripts like Hypothesis'. We deploy a patch or minor release for every pull request, but if that's undesirable you could run the deployment from a weekly "cron" job on Travis.
• Changelog management (for weekly rather than per-PR releases) can be automated with https://github.com/hawkowl/towncrier - there's a substitute for all of us 😄
• Xarray would need some novel scripts for uploading Github releases and driving the conda-forge feedstock. (Which Hypothesis would borrow in turn - it's lower priority for us but still nice to have)

In short, my advice is to be creative, and if release processes can't be automated - change them!