Stop having a "staging" environment

[dstufft]

Currently TestPyPI acts as two things, one is an environment for end users to upload copies of their software to to test their release processes and act as a sort of general sandbox for their testing and the other is a staging environment for changes to go through and get tested in.

I'm not going to include the entire post, but in @alex's recently blog post Don't have environments he makes the case that having explicit environments like this is a bad idea. He describes a scenario that truthfully has played out with PyPI legacy multiple times and although it has not happened for Warehouse, it's possible (likely?) that once we start having a large number of users hitting the code and actually caring about errors in production or broken features that we'll run into it here as well.

So in that vein, it may be a good idea to stop using TestPyPI as a staging environment and instead have master automatically deploy to both TestPyPI and PyPI. We currently have the ad hoc rule that master should be able to be deployed to PyPI at any time, but there is nothing really enforcing like.. actually deploying master to production immediately would.

If we do this, we'll need a few items to make it actually pratical:

• We'll need to enable Heroku Review Apps (however we'll need to ensure this is done safely). This will be made a bit more difficult since we tend not to use very many Heroku addons and instead bring our own services (often donated). Possibly we can just use free tier addons for this.
• A solution to database migrations (Automatically Run Database Migrations #661).

Doing this would mean that TestPyPI only exists for end users to use as a sandbox. We could possibly even get rid of TestPyPI completely if we implemented something like #726 which might (probably?) provide a better UX around testing releases in general.

[dstufft]

This is "done" in the sense that warehouse-staging.python.org is no longer staging but is a production environment and committing to master auto deploys to both it, and to warehouse.python.org. The other issues from above still need addressed.

[dstufft]

This is at best a new feature, and is more like an idea.

[brainwane]

@ewdurbin now that we've fixed #661, the only checkbox remaining from Donald's original post in this issue is the Heroku stuff. Is that still applicable? Can we close this?

[ewdurbin]

@brainwane this will require all kinds of other work to support closing the issue:

• Feature Deploys/Review Labs functionality
• Staged releases for users
• Doc purge of "testpypi" and "test.pypi.org"
• Possibly a deprecation window

It's a bit larger than just "turn it off" we can't do that until there's a way for folks to test workflows on PyPI itself as well as deployment changes internally.

[brainwane]

Thanks for detailing that and sorry for misunderstanding, @ewdurbin. Maybe sometime this spring (maybe at the sprints at PyCon?) we can break this out into sub-issues and spec them out.