Update to supporting Pydantic V2 #203

candleindark · 2023-11-20T04:00:30Z

This PR migrates Pydantic from V1 to V2. It closes #176.

This PR deprecates the following APIs:

dandischema.models.DandiBaseModel.json_dict
1. It can now be replaced by pydantic.BaseModel.model_dump(mode='json', exclude_none=True)
dandischema.models,DandiBaseModel.unvalidated
1. It can be replaced with pydantic.BaseModel.model_construct

Additional notes regarding changes entailed by Pydantic V2:

When using dandischema.models.CommonModel.repository in APIs which expect a str, use str() to convert the attribute to a str.
1. For more information, visit Url and Dsn types in pydantic.networks no longer inherit from str.
Pydantic V2 generates a slightly different JSON schema for the same Pydantic model compared to Pydantic V1. In particular, the JSON schemas for Optional fields now indicate that the value null is allowed. Because this change in behavior is incompatible with required tests for dandi-schema, dandischema.utils.TransitionalGenerateJsonSchema has been defined to revert the behavior of the JSON schema generation for Optional fields back to the behavior exhibited by Pydantic V1.
1. To generate the JSON schema for a model without the indication of the null value as for Pydantic V1, initiate the generation with Model.model_json_schema(schema_generator=TransitionalGenerateJsonSchema).
2. For more information about the use of a GenerateJsonSchema class, please checkout these examples.
Other changes in JSON schema generation in Pydantic V2 in comparison to Pydantic V1 that do not break current required tests for dandi-schema are the following:
1. "definitions" is now "$defs"
2. "default" is added for fields that have a default value
3. "acknowledgement" in CommonModel now has "description" instead of "descriptions" due to a typo correction I put in. (This is not a change entailed by Pydantic V2 but from a correction I put in the code.)
4. "maxLength" is no longer included in the JSON schema for a field of the type AnyHttpUrl. This is possibly due to the fact that all URL types in Pydantic V2 no longer inherit from str and there is no max length restriction.
5. For a field of a Literal type of a single value, the "type" key is no longer specified in the corresponding JSON schema. For example, the JSON schema for a field of the type Literal["Asset"] no longer contains the "type" key with the value of "string". This change affects fields such as BareAsset.schemaKey.
6. For an Enum subclass that has str values for corresponding choices, the new JSON schema will include a "type" key with the value of "string". This change affects pretty much all the Enum subclasses defined in this project.
7. The docstring is no longer inherited from a parent Pydantic model and used as the value for the "description" key in the JSON schema. For example, the JSON schema for Project no longer has the "description" key because its parent, Activity, has a docstring, """Information about the Project activity""".

Note: This PR contains the "minimum" changes needed to support Pydantic V2. Other improvements to the project enabled by Pydantic V2 will come as subsequent PRs.

TODOs:

Address the schema changes; possibilities:
- Check/eliminate divergences
- Boost schema version
Check WIP against dandi-cli in a PR
- Update for Pydantic 2.0 dandi-cli#1352
- Update for Pydantic v2 dandi-cli#1381
Check WIP against dandi-archive in a PR
- Migrate to pydantic 2.0 dandi-archive#1791

Note: bump-pydantic is still in beta

Replace its use with `pydantic.BaseModel.model_dump(mode='json', exclude_none=True)`

Its use can be replaced with `pydantic.BaseModel.model_construct`

yarikoptic · 2023-11-21T19:37:52Z

This PR removes the following APIs.

DandiBaseModel.json_dict

Its use can now be replaced by pydantic.BaseModel.model_dump(mode='json', exclude_none=True)

DandiBaseModel.unvalidated

Its use can be replaced with pydantic.BaseModel.model_construct

Even if for "practice" I think it might good to provide a proper deprecation cycle for those happen there is a tool using dandischema directly and not using any "pydantic specifics": please keep those in API, but make them raise DeprecationWarning and let's schedule to drop those entirely in e.g. 4-6 months after the entire migration to pydantic 2.0 "settles down".

candleindark · 2023-11-22T00:22:46Z

This PR removes the following APIs.

DandiBaseModel.json_dict

Its use can now be replaced by pydantic.BaseModel.model_dump(mode='json', exclude_none=True)

DandiBaseModel.unvalidated

Its use can be replaced with pydantic.BaseModel.model_construct

Even if for "practice" I think it might good to provide a proper deprecation cycle for those happen there is a tool using dandischema directly and not using any "pydantic specifics": please keep those in API, but make them raise DeprecationWarning and let's schedule to drop those entirely in e.g. 4-6 months after the entire migration to pydantic 2.0 "settles down".

@satra I think @yarikoptic's has a valid point. May be we should keep those APIs in place for a while instead of removing them now, as you said in Monday's meeting. I can keep those API in place by wrapping the new features provided by Pydantic 2.0.

satra · 2023-11-22T00:38:05Z

@candleindark - i'm fine with a deprecation cycle. however, if nobody is using them, i'm also ok with the decision to remove. we will be doing a significant change anyway with pydantic 2. if there are significant changes made to other pieces (cli, server), then let's remove those functions. if not, we can put them into deprecated mode.

But mark it as deprecated

Specifying the most compact separators to use in JSON serialization, `(",", ":")`, is no longer needed for they are the default separators to use for JSON serialization in Pydantic V2

…date_json()`

This is the implicit default in Pydantic V1

…lue`

…igest`

…rements`

…`BaseModel.model_json_schema`

…ave incompatible changes)

for more information, see https://pre-commit.ci

yarikoptic · 2024-01-16T18:00:10Z

current state: I boosted schema version to 0.7.0 and we need to review if any upgrades to the dict serialization are needed in migrate function where I added a placeholder in the code

The logic of the result is equivalent to the one before the modification

satra · 2024-01-18T18:44:03Z

@candleindark - could you please address the linting error?

@yarikoptic - i'm assuming we are releasing without the CLI changes? shall we release once the linting error is fixed?

jwodder · 2024-01-18T18:46:32Z

@satra No, we can't release yet, as the TODO item in this commit is still undone.

satra · 2024-01-18T18:50:39Z

there hasn't been any data model changes, has there? and i'm assuming the migrate tests all pass? the only time a migration code change is required is if we change the data model.

dandischema/metadata.py

jwodder · 2024-01-18T19:32:37Z

I've converted this PR to a draft in order to prevent accidental merging until we can confirm and are sure that all necessary changes have been made.

There is no change in the data model because of the Pydantic upgrade to V2

For dandi JSON schema 0.7.0 or above

dandischema/metadata.py

So that mypy doesn't complain

0.6.5 is more sensible since the changes in the schema are not breaking changes, and there is no change in the models and the JSON serialization of these models

candleindark · 2024-01-19T21:24:20Z

The test failed. It seems to be a github problem, but I don't have the option to re-run it.

candleindark · 2024-01-19T21:33:46Z

The test failed. It seems to be a github problem, but I don't have the option to re-run it.

Did someone just re-ran the test for me. It just re-ran and succeeded.

yarikoptic · 2024-01-22T15:19:11Z

I resolved merge conflict - trivial conflict in imports

Pydantic 2.0's JSON schema export no longer specifies a `type` field for the `schemaKey` property, and leaves it undefined instead. This change makes it so an undefined type also delegates it as a "basic schema", meaning it will not be rendered as its own tab in the meditor.

candleindark added 5 commits November 3, 2023 20:35

Change Pydantic dependency requirement to ~= 2.4

a6f3288

Merge branch 'master' into pydantic-v2

4395ad4

Migrate to Pydantic 2.0 using bump-pydantic 0.7.0

1decc23

Note: bump-pydantic is still in beta

Merge branch 'master' into with-bump-pydantic

14f742c

Update from BaseModel.__fields__ to BaseModel.model_fields

3b4ecf3

jwodder added the minor Increment the minor version when merged label Nov 20, 2023

candleindark added 2 commits November 20, 2023 19:18

Remove DandiBaseModel.json_dict

b211bfd

Replace its use with `pydantic.BaseModel.model_dump(mode='json', exclude_none=True)`

Remove DandiBaseModel.unvalidated

b697d76

Its use can be replaced with `pydantic.BaseModel.model_construct`

jwodder mentioned this pull request Nov 22, 2023

Overhaul models so that "unfinished" metadata can be represented without cheating Pydantic #205

Open

candleindark added 4 commits November 28, 2023 21:16

Keep DandiBaseModel.json_dict()

e0f75ce

But mark it as deprecated

Keeping DandiBaseModel.unvalidated()

92b5cb9

But mark it as deprecated

Replace the use of parse_obj_as() with TypeAdapter

739db75

Replace the use of update_forward_refs() with model_rebuild()

5528c3d

candleindark force-pushed the pydantic-v2 branch from 50a794c to 5528c3d Compare December 1, 2023 01:27

candleindark added 13 commits November 30, 2023 18:54

Replace the use of BaseModel.json() with BaseModel.model_dump_json()

ca3fff5

Specifying the most compact separators to use in JSON serialization, `(",", ":")`, is no longer needed for they are the default separators to use for JSON serialization in Pydantic V2

Replace the use of BaseModel.parse_raw() with `BaseModel.model_vali…

1d85790

…date_json()`

Add additional comments regarding BaseModel.model_rebuild()

5616209

Migrate the use of @validator() to @field_validator for schemaKey

e1adfb1

Specify default of None for PropertyValue.value

8c0e159

This is the implicit default in Pydantic V1

Migrate from @validator to @field_validator for `PropertyValue.va…

c21f918

…lue`

Migrate from @validator to @field_validator for BareAsset.digest

3b5fa3a

Migrate from @validator to @field_validator for `PublishedAsset.d…

acef6e1

…igest`

Migrate from @root_validator to @model_validator for Resource

c5b51bb

Migrate from @root_validator to @model_validator for `AccessRequi…

7fcb9be

…rements`

Use json_schema_extra to pass extra JSON Schema properties

b9604f5

Migrate the use of BaseModel.schema and BaseModel.schema_json to …

14b0d16

…`BaseModel.model_json_schema`

Migrate the use of Config class to model_config

fe85089

yarikoptic and others added 3 commits January 16, 2024 12:52

Boost dandi schema version to 0.7.0 (since exported jsonschema does h…

bddffd8

…ave incompatible changes)

Added placeholder for where to add migration

e36e003

[pre-commit.ci] auto fixes from pre-commit.com hooks

54315ae

for more information, see https://pre-commit.ci

Remove unneeded code in TransitionalGenerateJsonSchema.nullable_schema

7eb7272

The logic of the result is equivalent to the one before the modification

satra reviewed Jan 18, 2024

View reviewed changes

dandischema/metadata.py Outdated Show resolved Hide resolved

jwodder marked this pull request as draft January 18, 2024 19:32

candleindark added 2 commits January 19, 2024 12:02

Remove stub for migration to schema 0.7.0

51d4c53

There is no change in the data model because of the Pydantic upgrade to V2

Use JSON Schema validator for Draft 2020-12 in _validate_obj_json

b718dc2

For dandi JSON schema 0.7.0 or above

candleindark force-pushed the pydantic-v2 branch from 9c9085d to b718dc2 Compare January 19, 2024 20:21

candleindark commented Jan 19, 2024

View reviewed changes

dandischema/metadata.py Outdated Show resolved Hide resolved

candleindark added 2 commits January 19, 2024 12:33

Annotate two possible type of validator in _validate_obj_json

2e97166

So that mypy doesn't complain

Change the current version of the generated schema to 0.6.5 instead

889c775

0.6.5 is more sensible since the changes in the schema are not breaking changes, and there is no change in the models and the JSON serialization of these models

satra marked this pull request as ready for review January 19, 2024 21:18

Merge branch 'master' into pydantic-v2

2d48a09

yarikoptic merged commit 7af9f3a into dandi:master Jan 22, 2024
34 of 44 checks passed

candleindark deleted the pydantic-v2 branch January 26, 2024 22:01

This was referenced Jan 26, 2024

Allow dictionary representation of Dandiset to have extra attributes #218

Merged

Resolve remaining failures in #1823 dandi/dandi-archive#1838

Merged

yarikoptic mentioned this pull request Feb 2, 2024

Boost dandischema to 0.9.* series so we get support for pydantic 2.0 and schema 0.6.5 dandi/dandi-archive#1823

Merged

4 tasks

candleindark mentioned this pull request Feb 10, 2024

Customize JSON schema generation for Literal fields of Pydantic models #227

Closed

jwodder mentioned this pull request Aug 13, 2024

Declare typing_extensions dependency #248

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to supporting Pydantic V2 #203

Update to supporting Pydantic V2 #203

candleindark commented Nov 20, 2023 •

edited by jwodder

Loading

yarikoptic commented Nov 21, 2023

candleindark commented Nov 22, 2023 •

edited

Loading

satra commented Nov 22, 2023

yarikoptic commented Jan 16, 2024

satra commented Jan 18, 2024

jwodder commented Jan 18, 2024

satra commented Jan 18, 2024 •

edited

Loading

jwodder commented Jan 18, 2024

candleindark commented Jan 19, 2024

candleindark commented Jan 19, 2024

yarikoptic commented Jan 22, 2024

Update to supporting Pydantic V2 #203

Update to supporting Pydantic V2 #203

Conversation

candleindark commented Nov 20, 2023 • edited by jwodder Loading

yarikoptic commented Nov 21, 2023

candleindark commented Nov 22, 2023 • edited Loading

satra commented Nov 22, 2023

yarikoptic commented Jan 16, 2024

satra commented Jan 18, 2024

jwodder commented Jan 18, 2024

satra commented Jan 18, 2024 • edited Loading

jwodder commented Jan 18, 2024

candleindark commented Jan 19, 2024

candleindark commented Jan 19, 2024

yarikoptic commented Jan 22, 2024

candleindark commented Nov 20, 2023 •

edited by jwodder

Loading

candleindark commented Nov 22, 2023 •

edited

Loading

satra commented Jan 18, 2024 •

edited

Loading