RLS: 0.23.0

[jorisvandenbossche]

Release Checklist

This template can be copied into the RLS issue for a release and checked off as you go.

Pre-Release

• Open a pre-release issue and mention related parties. (A few weeks before release) with this template.
• Clear the bug queue
• Run an ASV: `cd asv_bench; asv continuous -f 1.1 $(git describe --abbrev=0)..HEAD``
• Review deprecations from previous releases here
• Review experimental features from previous releases here
• Clean up release.rst (contributors and stats will be done later)
• Clean up whatsnew.rst
• Run a full doc build and make sure everything is clean

  ./make.py clean; ./make.py; ./make.py zip_html;  ./make.py latex_forced
  

  • Check the output, check whatsnew and release.rst for formatting errors.
• run build_dist.sh, uninstall cython and make a test install of the tarball to ensure
  cython is not required for installation from tarball/pypi (now also part of travis scripts, but check).
• Ensure you have SSH access to pandas.pydata.org (@TomAugspurger or Andy Terrel should be able to get you an account).
• Ensure you have upload rights to PyPI (Wes or Jeff)
• Ensure you have commit rights on github.com/macPython/pandas-wheels/ and github.com/conda-forge/pandas-feedstock

Release Candidate

These are items that must be done for an RC. See the respective sections for details.

• Tag the Release
• Release on Github
• Build Wheels
• RC Conda Packages (waiting on conda-forge for full solution)
• Upload to PyPI
• Build the Docs
• Update the Website
• Announce

RC Conda Packages

Conda-forge doesn't support release candidates yet. We build them ourselves.

• Setup Windows Environment
• Setup Linux Environment

The Release

Final Pre-Release

To be done just before tagging the commit. This is optional for release-candidates.

• Call a hold on master in the pre-release issue before you start.
• Finalize the documentation by running the
  • Run scripts/announce.py and add to release.rst
  • Update the release date in release.rst and whatsnew/<version>.rst
• Commit the doc changes

Tag the Release

• Create an empty commit: git commit --allow-empty -m 'RLS: v0.20.0'
  • For Release-Candidates, it's -m 'RLS: v0.20.0.rc2'
• Create the tag: git tag -a v0.20.0 -m "Version 0.20.0"

Test the Release

Some local sanity tests before pushing, making the release official

• Build the release
  • Conda:

# build a conda package
conda build ./conda.recipe --numpy 1.11 --python 2.7 -q --output

# fresh conda env
conda create -n pandas_0.18.2_test python=2.7 pandas nose

# build and install into that env
conda install -n pandas_0.18.2_test pandas --use-local

# switch to that env & test
source activate pandas_0.18.2_test
import pandas as pd
pd.test()

• Pip

# build the wheel
python setup.py bdist_wheel

# activate another environment
workon pandas-test
pip uninstall -y pandas
pip install <path/to/wheel>
python -c "import pandas; pandas.test()"

• Build the docs. Check the version number and release notes

Push the Release

This is where things become final. No going back once the tagged commit is pushed

• Push the release
  • Major release: git push upstream master --follow-tags
  • Maintenance: git push upstream 0.20.x --follow-tags

Build Source Distribution

• git clean -xdf pandas && python setup.py cython && python setup.py sdist --formats=gztar (TODO: update build_dist.sh to handle maintenance branches)

Release on Github

• Draft a new release here
• If this is an RC, check the box saying it's a pre-release
• Upload the pandas-<version>.tar.gz from the previous step as a "binary"
• Generate the SHA256 for the release with openssl dgst -sha256 dist/pandas-<version>.tar.gz
  We use the SHA from our .tar.gz since the one auto-generated by Github isn't stable if the release page
  is later modified.

Build Binary Distributions

Windows wheels are built automatically by Christoph Gohlke.

• Open a PR for Mac and Linux wheels to update the BUILD_COMMIT here
• Final Release Only Open a PR for the pandas-feedstock
  • Update the version
  • Update the SHA256
  • Set the build number 5to 0

Download Binary Distributions

Conda-Forge is handled automatically. We handle wheels.

• Windows wheels from here
• Mac / Linux wheels with python scripts/fetch_wheels.py, from here

Upload to PyPI

Upload the source and wheels simultaneously. Uploading just the source may break some users'
workflows, if they trigger an update but don't have a C-complier

• Assuming you've downloaded everything to dist: twine upload dist/*
• You may want to hide the release until the docs are up and announcement is sent out

Build and Upload the Docs

• Build and upload

cd docs;
./make.py clean; ./make.py; ./make.py zip_html;  ./make.py latex_forced
python make.py upload_previous 0.20.2

• Update stable symlink: ln -sfn version/0.22.0 stable
• Update the minor symlink: ln -sfn 0.22.0 0.22

Update the Website

• clone the repo: https://github.com/pandas-dev/pandas-website.git
• major only: Move latest.rst to previous.rst
• Update latest.rst
• Update releases.json and pre_release.json (if RC, add to pre_release.json, else make it blank)
• Update _themes/pydata/layout.html to have the most recent minor release of each major version
• commit and push
• build: make html
• upload: make upload (this requires an SSH key for the docs server)
• Verify the cheatsheets at /Pandas_Cheet_Sheet.{pdf,pptx} are up to date (TODO: automate)
• Verify it updated (maybe in an incognito browser to avoid caching)

Announce

• Make sure everything is in place
  • pypi binaries for windows, mac wheels, source
  • docs built and published
  • website updated
• Send an ANN to the mailing list. include links to both release notes and whatsnew.
• send announcements to pydata@googlegroups.com, numpy-discussion@scipy.org, python-announce-list@python.org, scipy-user@scipy.org, pandas-dev@python.org

** Final Version**

• 0.20.3
• 0.20.2
• 0.20.1
• 0.19.2
• 0.19.1
• 0.19.0
• 0.18.1
• 0.18.0
• 0.17.1
• 0.17.0
• 0.16.2
• 0.16.1
• 0.16.0
• 0.15.2
• 0.15.1
• 0.15.0
• 0.14.1
• 0.14.0
• 0.13.1
• 0.13.0
• 0.11.0

** Release Candidates **

• 0.20.0rc1
• 0.18rc2
• 0.18rc1
• 0.17rc2
• 0.17rc1
• 0.16rc1
• 0.15rc1
• 0.14rc1
• 0.13rc1

Start the next release cycle

Only after a major release

When finishing a major release we have a few extra steps to ensure that
the development version is always ahead of the backports version.

• Create a backports branch off the release commmit, e.g. git checkout -b 0.20.x master
• Push that to upstream: git push -u upstream 0.20.x
• Switch back to master and make a new commit to start the next cycle
  • git checkout master
  • git commit --allow-empty -m 'DEV: Start 0.21 cycle'
  • git tag -a v0.21.0.dev0 -m 'Version 0.21.0 start'
  • git push upstream master --follow-tags
• Ensure that a whatsnew file is available for the next major and minor releases

Finish

• Close the Milestone on github
• Close the RLS issue

Miscellany I haven't gotten to yet.

Update documentation

• Get commits since last release:
  git log v0.18.1.. --format='%an#%s' | grep -v Merge > commits
• Include a summary by contributor in release notes (release.rst):
  cat commits | gawk -F '#' '{ print "- " $1 }' | sort | uniq
• Run release_stats.sh to get number of contributors and commits.
• In release notes/whatsnew, update release date.
• Update release notes (release.rst with highlights of whatsnew).

---

Tracking issue for 0.23.0

• Open issues/PRs: https://github.com/pandas-dev/pandas/milestones/0.23.0. It's still a big list, so we should do some cleaning on this to identify the important remaining ones
• What is needed for ExtensionArray before releasing: ExtensionArray meta-issue #19696 (let's keep that part of the discussion there)

(edited the milestone target date from end of March to end of April)

cc @pandas-dev/pandas-core

[TomAugspurger]

Thanks for opening this.

I'll go through the issues and mark ones that should be blockers, and push others.

[jorisvandenbossche]

Doing that as well. I created the 0.24.0 milestone, in case you wan't to push an issue, but not into the big pile of "next major release"

[jreback]

I would like to see 1 of the other existing types converted to EA before release. This will shake things down more. Likely the easiest is datetime w/tz.

[TomAugspurger]

Mmm I started on Interval this morning. Won't be too much work to finish that off. I can look into datetime w/ TZ as well.

…

On Fri, Mar 30, 2018 at 2:17 PM, Jeff Reback ***@***.***> wrote: I would like to see 1 of the other existing types converted to EA before release. This will shake things down more. Likely the easiest is datetime w/tz. — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#20531 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIt6fz0fmeR_K9tq7cU2DdtgfjYbKks5tjoS8gaJpZM4TAMfv> .

[jreback]

yeah the more the better, its going to surface other things. not necessarily will cause an api change, but that's the exercise (as it indeed might)

[TomAugspurger]

During the dev meeting we discussed postponing further ExtensionArray things until 0.24. Is that still OK with everyone? This would make the EA changes currently in master 100% backwards compatible. Moving Period / Interval to ExtensionArray will "breaking" since they aren't coerced to ndarrays of objects.

I'm still traveling this week, but will focus on reviewing PRs that are close, fixing blockers, and finalizing the EA docs (noting the experimental status). Then we can target a release candidate for Monday the 23, and plan to do the final release on April 27th or 30th?

[jreback]

yeah I think that schedule sounds good. I moved most things off of the milestone. So let's see what's left.

[jorisvandenbossche]

I moved most things off of the milestone. So let's see what's left.

In general, can we add a comment if an issue is moved in case it is labeled with things like "regression" or "blocker". Then at least there is a notification for it, instead of it happening silently.

Related, we can also make a temporary milestone 0.23.0rc to have a more focused one, without the need to remove regression/blockers (that we should still try to fix for 0.23.0) from the 0.23.0 milestone

[TomAugspurger]

I think we'll be able to do a release candidate tomorrow (the 27th).

[jreback]

it looks like statsmodels will be doing 0.9 soon
this is good - can finally merge that api change

[TomAugspurger]

Do we want to get a release out before PyCon?

I'm available to do the release stuff on Monday the 7th, which means we should do an RC today, but we still have a decent number of issues and PRs open.

Of the open PRs, I think the ones that may not be ready to merge immediately are

• API: categorical grouping will no longer return the cartesian product #20583
• Stop concat from attempting to sort mismatched columns by default #20613
• BUG: Fix freq setter for DatetimeIndex/TimedeltaIndex and deprecate for PeriodIndex #20772

These are close I think

• PERF: Fixed regression in Series(index=idx) constructor #20865
• PERF: Faster Series.__getattribute__ #20834
• BUG/DEPR: deprecate Categorical take default behaviour + fix Series[categorical].take #20841
• DOC: Fix read_hdf docstring #20874
• Emit warning for missing labels in Multiindex.loc[[...]] (and more) #20770

I'm going to push on the close ones over the next hour or two. I'd like to avoid major changes between the RC and final if possible.

[toobaz]

I guess in the next few hours we'll also decide about #20770 (the recently added docstring might need a review, but I guess it's better to squeeze that after the rc than the refactoring itself)

[TomAugspurger]

Sorry for missing #20770. Added to the close list.

[jreback]

i agree it would be nice to push a release before pycon, but I suspect we need a non-trivial RC period, this release is huge. Also we need to wait for statsmodels final I think as well (for #18341) and just to test. So I still lhave to fix for some comments on #20583 but if ok on the api, then ok to merge.

[jorisvandenbossche]

Also we need to wait for statsmodels final I think as well (for #18341) and just to test.

No need to wait on statsmodels IMO, we can just merge that PR in the next release (I mean, it would be nice to get rid of it, but it's not that it is urgent or release critical for us)

So I still lhave to fix for some comments on #20583 but if ok on the api, then ok to merge.

I still disagree on the deprecation part, but let's discuss there

[stonebig]

stastmodels-0.9.0rc1 is out

[TomAugspurger]

Down to

• Read from multiple <tbody> within a <table> #20891 (waiting on CI)
• Allow errors keyword for HDF IO Encoding Err Handling #20873 (waiting on CI)
• API: categorical grouping will no longer return the cartesian product #20583 (waiting on a self-+1 from @jreback)
• API: str.cat will align on Series #20347 (str_cat, @jreback or @jorisvandenbossche if you're -1 on that then speak up now)

May push #20885 till after the RC

Looking at #20844 now.

Does anyone have objections to be doing the RC this afternoon?

[stonebig]

shouldn't you mention you are compatible with Pyside2 ? (now that Qt makes an official statment http://blog.qt.io/blog/2018/04/13/qt-for-python-is-coming-to-a-computer-near-you/ )

A little marketing for the new officially supported Qt interface #20905

[TomAugspurger]

Just need a go / no-go on #20347, then I'm tagging the RC.

[jorisvandenbossche]

May push #20885 till after the RC

Yep, that one is not really critical for the RC (only a specific EA case), good enough for final release.

[jorisvandenbossche]

I added a comment on #20613, will now take a look at #20347 (ignored that the last days due to being too busy ...)

[TomAugspurger]

OK, tagging the RC now.

[jorisvandenbossche]

Thanks a lot, Tom!

[jreback]

thanks Tom! fantastic as usual

[TomAugspurger]

We didn't quite hit our deadline, but I think we're on track for a release this week sometime.

[jreback]

@TomAugspurger decks clear. just a minor comment on your PR. otherwise lgtm. (I will review a couple of PR's later, but only will merge if ready).

[TomAugspurger]

OK, thanks. I think we're all set for a release today.

I'll start on the process in ~2 hours.

[jreback]

yep just fixing pandas-wheels

i think new version of futures broke this

[TomAugspurger]

OK, tagging!

[TomAugspurger]

Tagged, pushed, and created the release.

FYI, @cgohlke not sure if you monitor our releases page, but the one at release at https://github.com/pandas-dev/pandas/releases/tag/v0.23.0 was initially messed up by me pressing enter, resulting in a v0 release at https://github.com/pandas-dev/pandas/releases/tag/v0. That can be ignored.

https://github.com/pandas-dev/pandas/releases/tag/v0.23.0 is the correct tag / release

[chris-b1]

Awesome work, thanks Tom!

[TomAugspurger]

All done. Going to clean up the release automation stuff I started in https://github.com/pandas-dev/pandas-release.

[jorisvandenbossche]

Also thanks a lot for all this infrastructure work!

[jorisvandenbossche]

Follow-up for 0.23.0: we already branched for 0.23.x ?

[TomAugspurger]

Follow-up for 0.23.0: we already branched for 0.23.x ?

Yes, I created 0.23.x off the 0.23.0 release commit. I think we have some depredations / API breaking changes that are ready to go into master.

[jorisvandenbossche]

OK, I just merged a first one with #18341

[TomAugspurger]

@pandas-dev/pandas-core reminder on the backporting workflow: Everything goes into master. We set the milestone to the correct version (e.g. 0.23.1) and we label items that need to be backported with the "Needs backport" label. When those are backported, we remove the label.

[jorisvandenbossche]

And to add to that: it goes into master in such a way that it is easy to backport. So if we decide a PR is for 0.23.1, the whatsnew the PR adds should be in v0.23.1.txt and not v0.24.0.txt, which means we decide before merging which version it targets.