Regenerate all Python tool lock-files (updating many) and update to pip_version = 24.0, for Py 3.12 support

[huonw]

This does three things to fix #20354 and give Pants Python 3.12 support by default:

• update the default [python].pip_version value to 24.0 (from 23.1.2), which is the current latest and is after 23.2, which is the first that supports Python 3.12
• run pants run build-support/bin/generate_builtin_lockfiles.py -- --all-python to update all tool lockfiles... this involves updating many tools, as a Big Bang, which is potentially unhelpful.
• to make this work, docformatter has to be restricted to the current version 1.4: 1.5.1 (the newest version that satisfies the old constraints) crashes on Pants itself (IndexError: list index out of range on pre-commit PyCQA/docformatter#151), while 1.7.5 (the newest version) also doesn't work on Pants (Update builtin docformatter to 1.7.5 #20498)

Here's a list of (as best I can tell), the changes in versions for the "main" requirement for each subsystem:

subsystem options scope	main requirement	old version	new version
helm-k8s-parser	hikaru	0.11.0b0	0.16.0b0
helm-post-renderer	yamlpath	3.7.0	3.8.1
autoflake	autoflake	2.0.1	2.1.1
bandit	bandit	1.7.4	1.7.5
black	black	23.1.0	23.3.0
pyupgrade	pyupgrade	3.3.1	3.3.2
ruff	ruff	0.2.1	0.2.2
yapf	yapf	0.32.0	0.40.2
coverage-py	coverage	7.2.1	7.2.7
debugpy	debugpy	1.6.6	1.6.7.post1
mypy	mypy	1.1.1	1.4.1
terraform-hcl2-parser	python-hcl2	4.3.0	4.3.2
yamllint	yamllint	1.29.0	1.32.0

There's numerous other transitive dependencies updated too, including in subsystems for which the main requirement hasn't changed and aren't listed above.

I haven't checked changelogs for any of these.

I've confirmed that running the reproducer from #20354 with this code now works. The commits are 'sensible'.

[cburroughs]

this involves updating many tools, as a Big Bang, which is potentially unhelpful.

In that it pulls in more updates than are strictly needed just for 3.12 compatibility?

[huonw]

Yeah, exactly.

[Eric-Arellano]

Thank you ❤️

[purajit]

I just noticed this because I was changing some settings and accidentally used the pants-provided lockfile instead of my own.

python-hcl2 should be pinned to <=4.3.0, since there's a huge host of parsing issues with 4.3.2. Basically everything here.

[huonw]

@purajit are you in a position to submit a fix along those lines? Thanks

[purajit]

@purajit are you in a position to submit a fix along those lines? Thanks

On it! Just wanted to be sure there wasn't an explicit reason to go to 4.3.2 besides just being newer. I also see there was a new version a couple weeks ago - going to check that out.

[purajit]

Made a PR ^