Make _PyDict_Pop() public

[scoder]

Feature or enhancement

Proposal:

It's used, it's reasonable, it's fast, it's already there. Just expose it instead of deleting it.

See #106320 and #108449

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-111262: Make PyDict_Pop() public #111263
• gh-111262: Add PyDict_Pop() function [with default value] #111939
• gh-111262: Add PyDict_Pop() function [without default value nor KeyError] #112028

[serhiy-storchaka]

We should first discuss the design.

The flaws of the current _PyDict_Pop():

• It is difficult to distinguish two cases: return value removed from the dict and returning the increfed default value. If the dict can contain arbitrary values, you need to create a special sentinel value.
• If you do not need the default value, you still have to decref it.

Alternative designs of PyDict_Pop() that do not raise KeyError.

1. PyObject *PyDict_Pop(PyObject *dict, PyObject *key)

   • Return NULL and set exception on error.
   • Return NULL and not set exception if the key is not found.
   • Return the removed value if the key is found.

   You have to use PyErr_Occurred() to distinguish between first two cases.

2. int PyDict_Pop(PyObject *dict, PyObject *key, PyObject **result)

   • Return -1, set *result to NULL and set exception on error.
   • Return 0, set *result to NULL and not set exception if the key is not found.
   • Return 1, set *result to the removed value if the key is found.

   You have not to use PyErr_Occurred(). You have to check the returning code or *result and *result to the default values after the call if you need it.

3. int PyDict_Pop(PyObject *dict, PyObject *key, PyObject **result)

   • Return -1, set *result to NULL and set exception on error.
   • Return 0, incref *result (which should be initialized with the default value) and not set exception if the key is not found.
   • Return 1, set *result to the removed value if the key is found.

   You have to always initialize *result before the call, but you can simply use *result after the call if you treat two last cases the same.

These variants do not raise KeyError. If you need KeyError, you have to raise it yourself. Or we can introduce new function which pops item and raises KeyError, or add a flag to raise KeyError. The last two variants have subvariants, how to treat result == NULL:

• Simply drop the removed value. It is convenient if PyDict_Pop() is used as a variant of PyDict_DelItem() that tolerates absent keys. It is the most of use cases in the CPython sources.
• Treat it as a signal to raise KeyError.

cc @rhettinger, @vstinner

[vstinner]

I like to return (-1, 0, 1). But I didn't get details of the API. The definition has not default argument? What is the behavior is default is NULL and the key does not exist.

Is it possible to call it with key=NULL?

[scoder]

Ok, I get the pattern. I'll add _PyDict_Pop() to #111162 so that we have an API until the bike-shedding has settled.

[vstinner]

Ok, I get the pattern. I'll add _PyDict_Pop() to #111162 so that we have an API until the bike-shedding has settled.

That sounds reasonable since 300+ private functions were removed, I don't expect that we can add public replacement quickly enough.

[vstinner]

After a long discussion, the int PyDict_Pop(PyObject *p, PyObject *key, PyObject **result) API won: there is no default value anymore. Thanks everyone for this interesting discussion.

Thanks @scoder for raising the need for such API.

[vstinner]

I wrote python/pythoncapi-compat#81 to add PyDict_Pop() and PyDict_PopString() to pythoncapi-compat: it calls _PyDict_Pop() on CPython >= 3.5.0b2, or call the dict.pop() method on older Cython versions and on PyPy.