Deploy to Jazzband package index from Travis CI #57

hugovk · 2020-09-30T09:38:02Z

For #18.

This will:

deploy to TestPyPI on merges to master (to ensure the release mechanism runs smoothly before release day)
deploy to production PyPI on tags
using https://github.com/pypa/gh-action-pypi-publish

TODO

Create API tokens (https://pypi.org/help/#apitoken) and save at GitHub Actions:

Create an API token called prettytable-ci at https://test.pypi.org/manage/account/token/
Save as test_pypi_password at https://github.com/jazzband/prettytable/settings/secrets/new
Create an API token called prettytable-ci at https://pypi.org/manage/account/token/
Save as pypi_password at https://github.com/jazzband/prettytable/settings/secrets/new

I can do (1) and (3), but don't have access to https://github.com/jazzband/prettytable/settings/secrets/new to do (2) or (4).

@flaper87 Do you have access to https://github.com/jazzband/prettytable/settings/secrets/new, or is this also a roadie-only task?

.github/workflows/deploy.yml

jezdez · 2020-10-02T08:08:03Z

To be clear, no Jazzband so far has used GitHub actions to do this, but always the recommended Travis CI integration. So there may be some debugging needed to get this right.

Regarding the use of the Test PyPI, I'm not sure if this is needed given the use of the Jazzband package index to do the releases. Project leads will need to review and confirm releases before they are pushed from the Jazzband package index to PyPI.

Co-authored-by: Jannis Leidel <jannis@leidel.info>

hugovk · 2020-10-02T08:59:28Z

To be clear, no Jazzband so far has used GitHub actions to do this, but always the recommended Travis CI integration. So there may be some debugging needed to get this right.

Shall we just switch over to Travis then?

There's not a huge gain for deploying from GHA (other than to be using the new thing). Especially if there's a lot on your plate we don't need to spend time debugging when Travis is know to deploy well.

If so, like this in .travis.yml?

deploy:
  provider: pypi
  user: jazzband
  server: https://jazzband.co/projects/prettytable/upload
  distributions: sdist bdist_wheel
  password:
    secure: svV4fYtodwW+iTyFOm5ISEfhVwcA+6vTskD3x6peznc40TdMV9Ek8nT3Q/NB4lCbXoUw2qR4H6uhLCjesnv/VvVk/qbitCyD8ySlgwOV5n7NzJs8lC8EYaHSjGQjatTwJAokfGVYkPawkI7HXDqtDggLUQBK+Ag8HDW+XBSbQIU=
  on:
    tags: true
    repo: jazzband/prettytable
    python: 3.8

(Based on https://github.com/jazzband/tablib/blob/5fa4496f9d3996c27f5252bba8cdd09e1fc278ef/.travis.yml#L28-L38)

Regarding the use of the Test PyPI, I'm not sure if this is needed given the use of the Jazzband package index to do the releases. Project leads will need to review and confirm releases before they are pushed from the Jazzband package index to PyPI.

Yeah, let's skip it when using Jazzband package index.

I've removed it from this PR.
Won't be needed if we use Travis.

jezdez · 2020-10-02T09:43:47Z

To be clear, no Jazzband so far has used GitHub actions to do this, but always the recommended Travis CI integration. So there may be some debugging needed to get this right.

Shall we just switch over to Travis then?

That would be preferable to me, yeah.

There's not a huge gain for deploying from GHA (other than to be using the new thing). Especially if there's a lot on your plate we don't need to spend time debugging when Travis is know to deploy well.

If so, like this in .travis.yml?
deploy:
  provider: pypi
  user: jazzband
  server: https://jazzband.co/projects/prettytable/upload
  distributions: sdist bdist_wheel
  password:
    secure: svV4fYtodwW+iTyFOm5ISEfhVwcA+6vTskD3x6peznc40TdMV9Ek8nT3Q/NB4lCbXoUw2qR4H6uhLCjesnv/VvVk/qbitCyD8ySlgwOV5n7NzJs8lC8EYaHSjGQjatTwJAokfGVYkPawkI7HXDqtDggLUQBK+Ag8HDW+XBSbQIU=
  on:
    tags: true
    repo: jazzband/prettytable
    python: 3.8

That should be right, I can update the upload secret once you've updated the PR.

(Based on https://github.com/jazzband/tablib/blob/5fa4496f9d3996c27f5252bba8cdd09e1fc278ef/.travis.yml#L28-L38)

Regarding the use of the Test PyPI, I'm not sure if this is needed given the use of the Jazzband package index to do the releases. Project leads will need to review and confirm releases before they are pushed from the Jazzband package index to PyPI.

Yeah, let's skip it when using Jazzband package index.
* I've removed it from this PR.

* Won't be needed if we use Travis.

Cool, thank you!

hugovk · 2020-10-02T11:25:50Z

PR updated!

ea0bcbc

hugovk · 2020-10-02T14:50:02Z

@jezdez Hi, I just pushed a tagged commit and the deploy failed:

Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 52.9kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 1 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 218kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 2 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 227kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 3 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 236kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 4 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 237kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 5 of 5
NOTE: Try --verbose to see response content.
HTTPError: 500 Internal Server Error from https://jazzband.co/projects/prettytable/upload
INTERNAL SERVER ERROR

https://travis-ci.org/github/jazzband/prettytable/jobs/732271558#L439

I've restarted the job and the same thing happened.

Before tagging, I'd made this tweak (d65f0d5) to .travis.yml so the deploy stage only runs for tags, but I don't think that's related.

jezdez · 2020-10-02T15:53:54Z

Thanks, something is wrong with the credentials, I think. Looking at the logs right now.

jezdez · 2020-10-02T15:58:36Z

TIL, this is a case of https://virtualandy.wordpress.com/2019/09/04/a-fix-for-operationalerror-psycopg2-operationalerror-ssl-error-decryption-failed-or-bad-record-mac/

jezdez · 2020-10-02T15:59:10Z

Glad you get to see this fail, despite having gotten countless releases from other Jazzband projects ;)

jezdez · 2020-10-02T16:24:16Z

@hugovk Ok, I've deployed a work-around until I get off of uWSGI. Can you retry the run on Travis?

hugovk · 2020-10-02T16:37:54Z

Also good we didn't throw GitHub Actions into the mix as another variable!

I restarted it and got the same thing:

Uploading distributions to https://jazzband.co/projects/prettytable/upload
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 45.5k/45.5k [00:01<00:00, 46.4kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 1 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 225kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 2 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 225kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 3 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 221kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 4 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 194kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 5 of 5
NOTE: Try --verbose to see response content.
HTTPError: 500 Internal Server Error from https://jazzband.co/projects/prettytable/upload
INTERNAL SERVER ERROR

https://travis-ci.org/github/jazzband/prettytable/jobs/732271558

jezdez · 2020-10-03T10:57:26Z

Okay, I got rid of uWSGI and deployed gunicorn instead, could you please try again?

hugovk · 2020-10-03T16:37:47Z

Same thing:

Uploading distributions to https://jazzband.co/projects/prettytable/upload
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|██████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 51.3kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 1 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 243kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 2 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 213kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 3 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 228kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 4 of 5
Uploading prettytable-1.0.0-py2.py3-none-any.whl
100%|███████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 258kB/s]
Received "500: INTERNAL SERVER ERROR" Package upload appears to have failed.  Retry 5 of 5
NOTE: Try --verbose to see response content.
HTTPError: 500 Internal Server Error from https://jazzband.co/projects/prettytable/upload
INTERNAL SERVER ERROR

https://travis-ci.org/github/jazzband/prettytable/jobs/732271558#L456

jezdez · 2020-10-03T20:15:19Z

Okay I think that fixed one problem, now to the other, I want to make sure the deploy key is correct, could you re-tag with the change I added to master please? 1.0.0...master I'm sorry for all of this kerfuffle, I'm embarrassed this is happening, but other releases have worked fine, I wonder it's a PEBKAC.

hugovk · 2020-10-03T20:37:05Z

No problem, these things happen!

Same thing again: https://travis-ci.org/github/jazzband/prettytable/jobs/732592087#L456

New tag: https://github.com/jazzband/prettytable/releases/tag/1.0.0
Tags last commit: 28592c0

jezdez · 2020-10-04T16:26:55Z

So I'm thinking the Authorization header isn't sent from Travis-CI for some reason, which kind of sucks. I think this is actually an issue with Travis-CI, e.g. travis-ci/dpl#1209

jezdez · 2020-10-04T16:33:01Z

@hugovk Soo, how do you feel to retrying the GH action integration?

If you're using the pypa/gh-action-pypi-publish action, repository_url needs to be https://jazzband.co/projects/prettytable/upload.

jezdez · 2020-10-04T16:35:58Z

@hugovk I've added the release key to the repo secrets as ${{ secrets.JAZZBAND_RELEASE_KEY }}

hugovk · 2020-10-04T18:43:19Z

Okay, let's try it!

It didn't work on the first try:

I created https://github.com/jazzband/prettytable/blob/master/.github/workflows/deploy.yml containing:

name: Deploy

on:
  push:
    branches:
      - master
  release:
    types:
      - published

jobs:
  build:
    if: github.repository == 'jazzband/prettytable'
    runs-on: ubuntu-20.04

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Cache
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: deploy-${{ hashFiles('**/setup.py') }}
          restore-keys: |
            deploy-
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Install dependencies
        run: |
          python -m pip install -U pip
          python -m pip install -U setuptools twine wheel

      - name: Build package
        run: |
          python setup.py --version
          python setup.py sdist --format=gztar bdist_wheel
          twine check dist/*

      - name: Publish package to Jazzband
        if: github.event.action == 'published'
        uses: pypa/gh-action-pypi-publish@master

        with:
          user: __token__
          password: ${{ secrets.JAZZBAND_RELEASE_KEY }}
          repository_url: https://jazzband.co/projects/prettytable/upload

Then deleted the old 1.0.0 tag and re-pushed it, and created a release from it at https://github.com/jazzband/prettytable/releases/tag/1.0.0 and it started the deploy:

Warning:  It looks like you are trying to use an API token to authenticate in the package index and your token value does not start with "pypi-" as it typically should. This may cause an authentication error. Please verify that you have copied your token properly if such an error occurs.
Checking dist/prettytable-1.0.0-py2.py3-none-any.whl: PASSED
Checking dist/prettytable-1.0.0.tar.gz: PASSED
Uploading distributions to jazzband.co/projects/prettytable/upload
Uploading prettytable-1.0.0-py2.py3-none-any.whl

  0%|          | 0.00/45.5k [00:00<?, ?B/s]
 18%|█▊        | 8.00k/45.5k [00:00<00:01, 20.3kB/s]
100%|██████████| 45.5k/45.5k [00:00<00:00, 64.9kB/s]
HTTPError: 401 Unauthorized from jazzband.co/projects/prettytable/upload
UNAUTHORIZED

https://github.com/jazzband/prettytable/runs/1206092655?check_suite_focus=true

That warning won't apply to the Jazzband PI.

Just to double check, is secrets.JAZZBAND_RELEASE_KEY meant to uppercase? We had lowercase earlier: 38745b6

jezdez · 2020-10-04T19:02:48Z

Yeah, for consistency sake upper case is correct, I've created it upper-case for certain. The issue is that the username needs to be jazzband, which stems from the fact that Jazzband's package index predates PyPI's choice of using __token__.

jezdez · 2020-10-04T19:04:05Z

If this works, I think I'll create a GitHub action to do this which we can use for all Jazzband projects, it's just so silly how I have to do this dance so often for Jazzband projects. And Travis has been acting up for a while now..

jezdez · 2020-10-04T19:14:59Z

Huzzah!! https://jazzband.co/projects/prettytable#1.0.0 🎊 🍾

jezdez · 2020-10-04T19:15:52Z

@hugovk You should have received an email explaining the rest of the process, don't forget to release both files per version, I'll work on the ability to release all files of a version soon.

hugovk · 2020-10-04T19:16:01Z

3f226d4 and success!

Checking dist/prettytable-1.0.0-py2.py3-none-any.whl: PASSED
Checking dist/prettytable-1.0.0.tar.gz: PASSED
Uploading distributions to jazzband.co/projects/prettytable/upload
Uploading prettytable-1.0.0-py2.py3-none-any.whl

  0%|          | 0.00/45.5k [00:00<?, ?B/s]
 18%|█▊        | 8.00k/45.5k [00:00<00:01, 25.2kB/s]
100%|██████████| 45.5k/45.5k [00:00<00:00, 68.1kB/s]
Uploading prettytable-1.0.0.tar.gz

  0%|          | 0.00/60.6k [00:00<?, ?B/s]
100%|██████████| 60.6k/60.6k [00:00<00:00, 208kB/s]

Thanks!

hugovk · 2020-10-04T19:26:00Z

@hugovk You should have received an email explaining the rest of the process, don't forget to release both files per version, I'll work on the ability to release all files of a version soon.

Yes, got the email (actually 3 emails: 2 for whl and one for sdist, and you got them too).

I released the first and got this error:

But the tar.gz was at PyPI: https://pypi.org/project/prettytable/#files

Then went back and did the .whl (no error) and now they're both there.

https://jazzband.co/projects/prettytable#1.0.0 shows a prettytable-1.0.0.tar.gz

Anyway, the main thing is we got a new release out after seven years! Thanks again!

jezdez · 2020-10-05T06:34:45Z

Yah, both are known issues that showed up over time and I had no time debugging them so far. Retrying the release of the tarball should make it work, this is something about Docker syncing the files not correctly via the volume release files are stored with.

jezdez · 2020-10-06T10:20:56Z

@hugovk I've fixed the issue of the releases and took the liberty to push the 1.0 tarball to PyPI since I didn't have another good way to reproduce this.

Long story short, the issue was related to PyPI's more aggressively caching the JSON API than in the past that the Jazzband website uses to confirm that the deploy to PyPI was successful. I'm using a timestamp in the URL now to cache-bust the call to the PyPI project JSON API.

hugovk · 2020-10-06T11:53:12Z

Thanks! I'll probably make an update release this week and will let you know how it goes!

hugovk · 2020-10-09T14:40:12Z

🚀 I just released 1.0.1, there was just a single tarball and a single wheel (as expected), and they both released to PyPI from Jazzband smoothly! 👍

jezdez · 2020-10-09T15:49:09Z

Thanks for letting me know @hugovk!

Deploy to PyPI and TestPyPI from GitHub Actions

9808619

hugovk added the release Deploy and release label Sep 30, 2020

hugovk mentioned this pull request Sep 30, 2020

New PyPI release? #18

Closed

jezdez requested changes Oct 2, 2020

View reviewed changes

.github/workflows/deploy.yml Outdated Show resolved Hide resolved

hugovk and others added 2 commits October 2, 2020 11:52

Use Jazzband package index

38745b6

Co-authored-by: Jannis Leidel <jannis@leidel.info>

Remove TestPyPI deploy

1929657

Deploy to Jazzband package index from Travis instead of GHA

ea0bcbc

hugovk changed the title ~~Deploy to PyPI and TestPyPI from GitHub Actions~~ Deploy to Jazzband package index from Travis CI Oct 2, 2020

jezdez merged commit ea0bcbc into jazzband:master Oct 2, 2020

hugovk deleted the deploy-to-pypi branch October 2, 2020 12:42

hugovk mentioned this pull request Oct 4, 2020

Update release checklist to deploy from GitHub Actions #61

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy to Jazzband package index from Travis CI #57

Deploy to Jazzband package index from Travis CI #57

hugovk commented Sep 30, 2020

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 3, 2020

hugovk commented Oct 3, 2020

jezdez commented Oct 3, 2020

hugovk commented Oct 3, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

hugovk commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

hugovk commented Oct 4, 2020

hugovk commented Oct 4, 2020

jezdez commented Oct 5, 2020

jezdez commented Oct 6, 2020

hugovk commented Oct 6, 2020

hugovk commented Oct 9, 2020

jezdez commented Oct 9, 2020

Deploy to Jazzband package index from Travis CI #57

Deploy to Jazzband package index from Travis CI #57

Conversation

hugovk commented Sep 30, 2020

TODO

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

jezdez commented Oct 2, 2020

hugovk commented Oct 2, 2020

jezdez commented Oct 3, 2020

hugovk commented Oct 3, 2020

jezdez commented Oct 3, 2020

hugovk commented Oct 3, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

hugovk commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

jezdez commented Oct 4, 2020

hugovk commented Oct 4, 2020

hugovk commented Oct 4, 2020

jezdez commented Oct 5, 2020

jezdez commented Oct 6, 2020

hugovk commented Oct 6, 2020

hugovk commented Oct 9, 2020

jezdez commented Oct 9, 2020