Argument Clinic: add support for deprecating positional use of parameters

[erlend-aasland]

Feature or enhancement

Suggesting to add syntax for producing deprecation warnings for positional use of optional parameters.

I made a proof-of-concept patch for this where I introduced a new special symbol x. Example use:

/*[clinic input]
mod.func
    stuff: object
    /
    x
    optarg: int = 128

This will then generate code that emits a DeprecationWarning if optarg is passed as a positional argument, but not if it is passed as a keyword.

We can use this feature to introduce deprecation warnings for parameters that will become keyword-only in future releases. When the deprecation period is done, we can simply replace the x with *, to really make the optional params keyword-only.

Pitch

Quoting @serhiy-storchaka, in issue #93057:

It is recommended to make optional rarely used arguments keyword-only. I do not think it will break much code, but we need a deprecation period for this.

The problem is that sqlite3.connect() was converted to Argument Clinic, and it was much easier to add deprecation warning in the old code.

Previous discussion

• Make optional parameters of sqlite3.connect() keyword-only #93057

Linked PRs

• gh-95065: Add Argument Clinic support for deprecating positional use of parameters #95151
• gh-95065: Argument Clinic: Pretty-print long C strings in generated code #107712
• gh-95065: PoC: Argument Clinic: Support multiple '* [from ...]' deprecations #107742
• gh-95065: Make Argument Clinic append deprecation warnings to docstrings #107745
• gh-95065: Argument Clinic: Add comment to preprocessor warning code #107766
• gh-95065: Argument Clinic: Add functional tests of deprecated positionals #107768
• gh-95065: Produce nicer deprecation messages in Argument Clinic #107808
• gh-95065, gh-107704: Argument Clinic: support multiple '/ [from ...]' and '* [from ...]' markers #108132

[erlend-aasland]

If there is sufficient support for this feature, I'll create a PR.

[erlend-aasland]

If there is sufficient support for this feature, I'll create a PR.

Two Three thumbs up; I'll create the PR 😎

[zware]

I'd suggest to go full warnings._deprecated on it, with an error if the specified removal version has reached beta without the change.

[erlend-aasland]

I'd suggest to go full warnings._deprecated on it, with an error if the specified removal version has reached beta without the change.

Maybe I'm misunderstanding you, but that would imply extending the new syntax with optional deprecated version and removed version fields, right?

[serhiy-storchaka]

I think it is great idea, although I would like to use something more distinguishable and searchable than just x.

We should consider more general problem, when we want to change parameters in incompatible way.

1. Turn positional-or-keyword parameters into keyword-only parameters. A directive which means "the following positional-or-keyword parameters will be keyword-only parameters in future".
2. Turn positional-or-keyword parameters into positional-only parameters. A directive which means "the preceding positional-or-keyword parameters will be positional-only parameters in future".
3. Remove parameter. For some converters we can use NULL or something like to distinguish absent argument, but for others all values after conversion are correct, so we need a special directive for this. When remove positional-only or positional-or-keyword parameter, all following parameters should become keyword-only parameters.
4. Rename positional-or-keyword or keyword-only parameters. In meantime the function should accept both old and new keyword names, but raise an exception if both are passed or if the argument is passed as positional and keyword. For example see Wrong keyword parameter name in regex pattern methods #64482.

There may be other possible incompatible changes, but these four are common.

[erlend-aasland]

I think it is great idea, although I would like to use something more distinguishable and searchable than just x.

Thanks for the support! I'm up for any other symbol; x was just the first thing that popped up in my head when I did the first proof-of-concept implementation. OTOH, * and / are not very searchable either.

We should consider more general problem, when we want to change parameters in incompatible way. [...]

There may be other possible incompatible changes, but these four are common.

+1 However, I think we should create separate issues for each feature :)

[hauntsaninja]

Thanks for suggesting this! This would be nice, e.g. for #56166

A quick bikeshed, do we need to use a new symbol? Something like the following seems pretty readable:

posonlyarg
/
futureposonlyarg
/-in-future
posorkwarg
*-in-3.12
futurekwonlyarg
*
kwonlyarg

Over here *-in-3.12 is a suggestion for how we could specify we want warnings._deprecated semantics.

[erlend-aasland]

do we need to use a new symbol?

I don't know; I just picked that path, because it was very easy to implement as a proof of concept :)

Regarding the proposed extended format: I'd replace the hyphens with single whitespace to make it more readable.

[erlend-aasland]

cc. @colorfulappl: this might be of interest to you.

[erlend-aasland]

Expanding on @hauntsaninja's bikeshedding in #95065 (comment):

All of these should mean: "positional use is deprecated; param will be keyword only starting from 3.14":

• *-in-3.14 (Shantanu's suggestion)
• *-only-in-3.14 (more explicit)
• * keyword-only in 3.14 (even more explicit, and why not use spaces?)
• *-3.14 (short, but slightly confusing?)

[erlend-aasland]

Example clinic code using the second suggestion from #95065 (comment):

/*[clinic input]
mymeth

    a: object
    b: object
    c: int = 5
    d: bool = True
    e: float = 0.5

[clinic start generated code]*/

Now, let's deprecate stuff:

/*[clinic input]
mymeth

    a: object
    b: object
    *-only-in-3.16
    c: int = 5
    d: bool = True
    *-only-in-3.14
    e: float = 0.5

[clinic start generated code]*/

[serhiy-storchaka]

I have no preferences of used words, but if use spaces, I would prefer to use also some parenthesis. * (only in 3.14) or * [only in 3.14] or [only in 3.14] *.

[erlend-aasland]

Yeah, the parentheses look nice.

Also, I think we should not allow deprecations without a deadline.

[AA-Turner]

Some additional suggestions:

• * [from 3.14] -- i.e. the following parameters are keyword-only from Python 3.14 onwards.

If the in form were used, we could say:

• * [in future] to allow an open deadline

Or, combining two approaches:

• * futureposonlyarg (from 3.14) for a deprecation with date
• * futureposonlyarg for a deprecation without a date

Not meant to be fully fleshed out, but hopefully might provoke other ideas!

A

[erlend-aasland]

I updated #95151 using @AA-Turner's suggested format (simply since I like that best, personally 😄). For now, I've keep the feature dead simple; there is no support for multiple sets of deprecations.

IMO, it would be wise to add a simple variant of the feature first, try it out, and then add support for multiple deprecation variants later.

@zware, I added preprocessor whining if clinic code has not been updated yet. Hopefully it is not too whiny :) (I guess #error is too harsh)

[erlend-aasland]

Suggestions for follow-up PRs after #95151 lands:

• Move the clinic.test.c tests to Modules/_testclinic.c, in order to test the emitted warnings
• Simplify the generated preprocessor code, so that the warning messages for the parameters are consolidated into one message, and only one bulk of preprocessor code is generated
• Add tests for preprocessor warnings/errors
• Support multiple [from ...] lines (for example, deprecate 2 params for Python x.y, and 3 more params for Python x.y+1) Abandoned. See gh-95065: PoC: Argument Clinic: Support multiple '* [from ...]' deprecations #107742 for more info.
• Pretty-print deprecation warning for some corner cases
• Pretty-print generated preprocessor code (wrap lines, etc.)

[erlend-aasland]

For multiple deprecation layers, I believe we want:

• one consolidated deprecation message a la:

  Passing more than 1 positional arguments to _sqlite3.Connection.init() is deprecated. Parameters 'timeout', 'detect_types' and 'isolation_level' will become keyword-only parameters in Python 3.14, parameter 'check_same_thread' will become a keyword-only parameter in Python 3.15, and parameters 'factory', 'cached_statements' and 'uri' will become keyword-only parameters in Python 3.16

• generate multiple blocks of preprocessor code, one per deprecation group

[AA-Turner]

One potential downside of a consolidated deprecation message is that it is harder to filter on parts -- e.g. if you have migrated all usages of parameter X to keyword, but not Y, you could filter out the Y warnings in your test suite, but ensure that any new X warnings cause test failures.

I don't know how realistic this would be though, as you might just migrate all parameters at once rather than in stages.

A

[erlend-aasland]

One potential downside of a consolidated deprecation message is that it is harder to filter on parts -- e.g. if you have migrated all usages of parameter X to keyword, but not Y, you could filter out the Y warnings in your test suite, but ensure that any new X warnings cause test failures.

I don't know how realistic this would be though, as you might just migrate all parameters at once rather than in stages.

See gh-107742 for a rough proof-of-concept; I'm not sure this feature is worth it. At the moment, I think we should play with what we've got. If the need arises, we can consider expanding the feature so multiple deprecation layers/groups are supported.

[erlend-aasland]

One thing I thought of that could be nice, though, is to append the deprecation warning to the docstring.

See gh-107745 for a suggestion on how to do this.

[serhiy-storchaka]

@erlend-aasland, do you mind to create issues for other two problems mentioned in #95065 (comment)? I could do them, but I am not sure about syntax.

[erlend-aasland]

@erlend-aasland, do you mind to create issues for other two problems mentioned in #95065 (comment)? I could do them, but I am not sure about syntax.

I'll get to it right away.

[erlend-aasland]

I think we can close this now. The only thing missing is the preprocessor tests; I'm not sure it is worth it to add all the boilerplate code needed to get such tests up and running. If we feel it is needed, we can create a follow-up issue for that particular item, but I don't think it should keep blocking the resolution of this issue.

[erlend-aasland]

Thanks again, to everyone involved!