Pin python version for failing cron jobs

[mvandenburgh]

Some cron jobs are failing due to an image rebuild that happened recently. Since we don't usually pin python versions, some images got rebuilt with python 3.12, which introduced some compatability issues with some of our pinned dependencies. This PR pins all of these images to python 3.11 as a stopgap until we can sort out how to deal with upgrading dependencies.

(original description)
This upgrades all python dependencies in requirements.txt for our python cron jobs. Some of these are failing now due to incompatibilities between Python 3.12 and the older dependencies that are pinned.

[mvandenburgh]

This is another instance of the issue mentioned in #618.

[scottwittenburg]

I'm curious how you picked the versions to pin, and whether you have any idea if the new versions have breaking api changes.

[mvandenburgh]

I'm curious how you picked the versions to pin, and whether you have any idea if the new versions have breaking api changes.

I used https://github.com/simion/pip-upgrader to do it, which is a tool that just bumps all the dependencies in a requirements.txt file to the latest version.

As far as breaking changes, I'm not sure. In general I'm not sure the best way to handle upgrades like this; what I was thinking is we just do the upgrade, fully expecting that there may be errors, and just keep an eye on sentry and make any fixes as we encounter them. I'm definitely open to other ideas though.

[scottwittenburg]

I used https://github.com/simion/pip-upgrader to do it, which is a tool that just bumps all the dependencies in a requirements.txt file to the latest version.

Oh, that looks interesting, though unmaintained for a couple years. I wonder if I should be double-checking what it produced. Like is fast-api really at v0.104.x? Some of them kind of caught my eye.

we just do the upgrade, fully expecting that there may be errors, and just keep an eye on sentry and make any fixes as we encounter them

I think that could be ok, assuming sentry is hooked up to all those, I noticed you've been adding to it some or maybe all our python services recently.

This makes me think of our staging environment, but I have no idea if these pieces are running there, or if not, whether it makes sense to run them there. Also, some cron job that runs every 12 hours, would we just wait to see a successful pod record in staging from all the affected services?

I'm just rambling I guess, maybe @zackgalbreath wants to chime in though.

[mvandenburgh]

I used simion/pip-upgrader to do it, which is a tool that just bumps all the dependencies in a requirements.txt file to the latest version.

Oh, that looks interesting, though unmaintained for a couple years. I wonder if I should be double-checking what it produced. Like is fast-api really at v0.104.x? Some of them kind of caught my eye.

Yup! https://github.com/tiangolo/fastapi/releases/tag/0.104.0

And yeah it's not really maintained, but all the tool really does is parse requirements.txt => grab latest version of each package from PyPI => write those to requirements.txt. I can do a quick double check to make sure it's all valid.

I think that could be ok, assuming sentry is hooked up to all those, I noticed you've been adding to it some or maybe all our python services recently.

Yes, all of the images affected here are hooked up to sentry now.

This makes me think of our staging environment, but I have no idea if these pieces are running there, or if not, whether it makes sense to run them there. Also, some cron job that runs every 12 hours, would we just wait to see a successful pod record in staging from all the affected services?

Sadly, none of these run in staging currently. We should hook those up at some point. But you have a good point about cron jobs, I suppose we would just have to wait (unless, it's something we can invoke manually just to test. Like I think we could do that for the gitlab-api-scrape without any issues, but obviously something like the snapshot-release job we wouldn't want to run).

The only other option I can think of to mitigate this issue is adding tests (with stuff like github/gitlap mocked properly) like we have for the sync script. Or we could of course just pin to an older Python version and leave the existing pinned versions, but that would just delay this for another day.

[mvandenburgh]

@scottwittenburg in light of the points you brought up, I tried a different approach - I backed out the requirements.txt updates, and just pinned the Python version to 3.11 so that the old pinned dependencies can still work. This will at least get these jobs running again, and maybe we can discuss the dependency pinning issue separately. Does that sound ok to you?

[scottwittenburg]

his will at least get these jobs running again, and maybe we can discuss the dependency pinning issue separately. Does that sound ok to you?

Well, I'm sort of bummed to be the blocker of forward progress, but yes, this sounds ok to me. At least get the jobs running again for now, then we can discuss the right long-term approach without that siren going off in our ears.

[scottwittenburg]

You maybe are already doing this, but since you merged #686, do you want to rebase this on main now and make sure these images build?

Also, maybe you can update the description since the approach has changed since you wrote the original one?