chore(ingest): start working on pydantic v2 support

[hsheth2]

Note that this does not fully enabled pydantic v2 compatibility yet (#8829). However, it lays the groundwork for doing so in the future.

Given the large surface area of changes, I want to merge this quickly so that we can avoid merge conflicts.

• Ran a dry-run version of the bump-pydantic codemod.
• Adds skip_on_failures to non-pre root validators. This will reduce the quality of our error messages a bit.
• Makes None the explicit default for fields that are optional
• Adds type annotations for inherited config fields, as well as for a few fields that were missing them.

Remaining TODOs

• Make sure we covered all the Optional[x] fields when adding the default of None.
• Migrate the data contracts subcommand - it uses root models, which don't really translate cleanly between pydantic 1 and 2.
• Figure out how to replicate parse_obj_allow_extras (and more broadly, get all the tests to pass with pydantic 2)

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[treff7es]

Is this intentionally commented out?

[hsheth2]

yes - mainly as a marker for how to type this function in the future