bpo-41842: Add a unregister function in _codecs module #22360

shihai1991 · 2020-09-22T13:18:02Z

Add a _codecs.unregister() in _codecs module to unregister search function.

https://bugs.python.org/issue41842

shihai1991 · 2020-09-22T14:53:20Z

Pls take a look if you have free times, thanks. @vstinner @serhiy-storchaka

vstinner

Can you please add an entry in the Doc/whatsnew/3.10.rst document? You have to add a news "codecs" section near:
https://docs.python.org/dev/whatsnew/3.10.html#improved-modules

Doc/library/codecs.rst

Misc/NEWS.d/next/Library/2020-09-22-20-54-18.bpo-39337.zX2e5M.rst

Modules/_codecsmodule.c

Python/codecs.c

vstinner · 2020-09-23T12:48:48Z

It would be nice to use this new function to remove an old side effect of test_codecs: codecs.register(_get_test_codec) registers a search function which is never unregistered.

Maybe ExceptionChainingTest.setUp() method should register the search function and use self.addCleanup(codecs.unregister, _get_test_codec)?

vstinner · 2020-09-23T12:51:10Z

See this comment of test_codecs.py:

        # There's no way to unregister a codec search function, so we just
        # ensure we render this one fairly harmless after the test
        # case finishes by using the test case repr as the codec name
        # The codecs module normalizes codec names, although this doesn't
        # appear to be formally documented...
        # We also make sure we use a truly unique id for the custom codec
        # to avoid issues with the codec cache when running these tests
        # multiple times (e.g. when hunting for refleaks)

And https://bugs.python.org/issue22166

vstinner · 2020-09-23T12:52:24Z

Old email requesting codecs.unregister() function: https://mail.python.org/pipermail/python-dev/2011-September/113588.html

vstinner · 2020-09-23T12:58:43Z

https://bugs.python.org/issue39337 is not directly related to codecs.unregister().

@shihai1991: I created https://bugs.python.org/issue41842 "Add codecs.unregister() to unregister a codec search function": please use this bpo for your PR.

Python/codecs.c

shihai1991 · 2020-09-23T14:49:31Z

Maybe ExceptionChainingTest.setUp() method should register the search function and use self.addCleanup(codecs.unregister, _get_test_codec)?

+1， it is in my TODO list :)

vstinner · 2020-09-23T14:50:25Z

Updating test_codecs can be done in a second following PR if you prefer.

shihai1991 · 2020-09-23T15:10:42Z

https://bugs.python.org/issue39337 is not directly related to codecs.unregister().

@shihai1991: I created https://bugs.python.org/issue41842 "Add codecs.unregister() to unregister a codec search function": please use this bpo for your PR.

copy that, this bpo have been used in this PR.

Doc/library/codecs.rst

Doc/whatsnew/3.10.rst

Include/codecs.h

Lib/test/test_codecs.py

Misc/NEWS.d/next/Library/2020-09-23-22-52-24.bpo-41842.lIuhC9.rst

Modules/_codecsmodule.c

Python/codecs.c

Include/codecs.h

Lib/test/test_codecs.py

Python/codecs.c

Doc/c-api/codec.rst

Doc/whatsnew/3.10.rst

Include/codecs.h

Lib/test/test_codecs.py

* origin/master: (27 commits) bpo-41428: Fix compiler warnings in unionobject.c (pythonGH-22388) bpo-41654: Fix compiler warning in MemoryError_dealloc() (pythonGH-22387) bpo-41833: threading.Thread now uses the target name (pythonGH-22357) bpo-30155: Add macros to get tzinfo from datetime instances (pythonGH-21633) bpo-33822: Update IDLE section of What's New 3.8 (pythonGH-22383) bpo-41844: Add IDLE section to What's New 3.9 (GN-22382) bpo-41841: Prepare IDLE News for 3.10 (pythonGH-22379) bpo-37779 : Add information about the overriding behavior of ConfigParser.read (pythonGH-15177) bpo-40170: Use inline _PyType_HasFeature() function (pythonGH-22375) bpo-40941: Fix stackdepth compiler warnings (pythonGH-22377) bpo-40941: Fix fold_tuple_on_constants() compiler warnings (pythonGH-22378) bpo-40521: Fix PyUnicode_InternInPlace() (pythonGH-22376) bpo-41834: Remove _Py_CheckRecursionLimit variable (pythonGH-22359) bpo-1635741, unicodedata: add ucd_type parameter to UCD_Check() macro (pythonGH-22328) bpo-1635741: Port _lsprof extension to multi-phase init (PEP 489) (pythonGH-22220) bpo-41513: Improve order of adding fractional values. Improve variable names. (pythonGH-22368) bpo-41816: `StrEnum.__str__` is `str.__str__` (pythonGH-22362) bpo-35764: Rewrite the IDLE Calltips doc section (pythonGH-22363) bpo-41810: Reintroduce `types.EllipsisType`, `.NoneType` & `.NotImplementedType` (pythonGH-22336) bpo-41602: raise SIGINT exit code on KeyboardInterrupt from pymain_run_module (python#21956) ...

vstinner

LGTM. I just requested a bunch of minor edits.

I leave the PR open a few days if @malemburg wants to have a look.

cc @serhiy-storchaka

Doc/whatsnew/3.10.rst

Lib/test/test_codecs.py

Python/codecs.c

Doc/whatsnew/3.10.rst

ZackerySpytz · 2020-09-26T10:34:16Z

Python/codecs.c

+        return 0;
+    }
+
+    Py_ssize_t n = PyList_Size(codec_search_path);


Please use the macro PyList_GET_SIZE() instead of PyList_Size(), as codec_search_path is guaranteed to be a list.

While it's a list right now, its type can change tomorrow.

@shihai1991: If you modify the code to use fast macros, please add the following assertion before using it:

assert(PyList_CheckExact(interp->codec_search_path));

@vstinner Copy that, I have already updated it.
But I have a question. assert(PyList_Check(op) in _PyList_CAST() is not good enough?

Python/codecs.c

Misc/NEWS.d/next/Library/2020-09-23-22-52-24.bpo-41842.lIuhC9.rst

vstinner · 2020-09-28T21:41:25Z

Merged, thank you @shihai1991!

shihai1991 · 2020-09-29T03:57:07Z

Thanks for everyone's review: ).

* origin/master: (113 commits) bpo-41773: Raise exception for non-finite weights in random.choices(). (pythonGH-22441) bpo-41873: Add vectorcall for float() (pythonGH-22432) bpo-41861: Convert _sqlite3 PrepareProtocolType to heap type (pythonGH-22428) bpo-41842: Add codecs.unregister() function (pythonGH-22360) bpo-41875: Use __builtin_unreachable when possible (pythonGH-22433) bpo-40105: ZipFile truncate in append mode with shorter comment (pythonGH-19337) bpo-41870: Use PEP 590 vectorcall to speed up bool() (pythonGH-22427) [doc] Leverage the fact that the actual types can now be indexed for typing (pythonGH-22340) bpo-41861: Convert _sqlite3 cache and node static types to heap types (pythonGH-22417) bpo-41858: Clarify line in optparse doc (pythonGH-22407) Revert "Fix all Python Cookbook links (python#22205)" (pythonGH-22424) bpo-1635741: Port _bisect module to multi-phase init (pythonGH-22415) bpo-41428: Fix compiler warning in unionobject.c (pythonGH-22416) Fix logging error message (pythonGH-22410) bpo-39934: Account for control blocks in 'except' in compiler. (pythonGH-22395) bpo-41775: Make 'IDLE Shell' the shell title (python#22399) bpo-41428: Fix compiler warnings in unionobject.c (pythonGH-22388) bpo-41654: Fix compiler warning in MemoryError_dealloc() (pythonGH-22387) bpo-41833: threading.Thread now uses the target name (pythonGH-22357) bpo-30155: Add macros to get tzinfo from datetime instances (pythonGH-21633) ...

Add codecs.unregister() and PyCodec_Unregister() functions to unregister a codec search function.

the-knights-who-say-ni added the CLA signed label Sep 22, 2020

bedevere-bot added the awaiting review label Sep 22, 2020

shihai1991 changed the title ~~Add a unregister function in _codecs module~~ [WIP] Add a unregister function in _codecs module Sep 22, 2020

shihai1991 force-pushed the bpo_39337_new_3 branch from 0692346 to 03645a7 Compare September 22, 2020 13:23

add a _codecs.unregister in _codecs module

fefda70

shihai1991 force-pushed the bpo_39337_new_3 branch from 03645a7 to fefda70 Compare September 22, 2020 13:57

shihai1991 changed the title ~~[WIP] Add a unregister function in _codecs module~~ bpo-39337: Add a unregister function in _codecs module Sep 22, 2020

vstinner reviewed Sep 23, 2020

View reviewed changes

Python/codecs.c Show resolved Hide resolved

apply victor's comment

f7775c0

shihai1991 changed the title ~~bpo-39337: Add a unregister function in _codecs module~~ bpo-41842: Add a unregister function in _codecs module Sep 23, 2020

vstinner reviewed Sep 23, 2020

View reviewed changes

apply victor's comment

71b624e

shihai1991 requested a review from vstinner September 23, 2020 17:38

vstinner reviewed Sep 23, 2020

View reviewed changes

Include/codecs.h Show resolved Hide resolved

Include/codecs.h Show resolved Hide resolved

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

Python/codecs.c Outdated Show resolved Hide resolved

shihai1991 added 2 commits September 24, 2020 12:45

apply victor's comment

d186726

retrigger checks

f8e912b

vstinner reviewed Sep 24, 2020

View reviewed changes

shihai1991 added 4 commits September 24, 2020 19:27

apply victor's comment

40666fc

update doc

3257b6c

update doc

a493a35

update doc

db4631e

vstinner approved these changes Sep 24, 2020

View reviewed changes

Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved

Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved

Lib/test/test_codecs.py Show resolved Hide resolved

Lib/test/test_codecs.py Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Sep 24, 2020

apply victor's comment

5efff22

corona10 reviewed Sep 24, 2020

View reviewed changes

Python/codecs.c Show resolved Hide resolved

apply Dong-hee Na's comment

f18cd4f

vstinner reviewed Sep 26, 2020

View reviewed changes

Python/codecs.c Outdated Show resolved Hide resolved

ZackerySpytz reviewed Sep 26, 2020

View reviewed changes

shihai1991 added 2 commits September 27, 2020 00:02

apply victor and Zackery's comment

8f6dc4d

apply victor's comment

4308ea8

vstinner reviewed Sep 27, 2020

View reviewed changes

Misc/NEWS.d/next/Library/2020-09-23-22-52-24.bpo-41842.lIuhC9.rst Outdated Show resolved Hide resolved

apply victor's comment

c9a0c99

taleinat mentioned this pull request Sep 27, 2020

bpo-39337: Add a test case for normalizing of codec names #19069

Merged

vstinner merged commit d332e7b into python:master Sep 28, 2020

bedevere-bot removed the awaiting merge label Sep 28, 2020

xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020

bpo-41842: Add codecs.unregister() function (pythonGH-22360)

e3625f2

Add codecs.unregister() and PyCodec_Unregister() functions to unregister a codec search function.

lancelote mentioned this pull request May 27, 2021

Add codecs.unregister() for Python 3.10 python/typeshed#5539

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-41842: Add a unregister function in _codecs module #22360

bpo-41842: Add a unregister function in _codecs module #22360

shihai1991 commented Sep 22, 2020 •

edited

Loading

shihai1991 commented Sep 22, 2020 •

edited

Loading

vstinner left a comment

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

shihai1991 commented Sep 23, 2020

vstinner commented Sep 23, 2020

shihai1991 commented Sep 23, 2020

vstinner left a comment

ZackerySpytz Sep 26, 2020

vstinner Sep 26, 2020

shihai1991 Sep 26, 2020

vstinner commented Sep 28, 2020

shihai1991 commented Sep 29, 2020 •

edited

Loading

bpo-41842: Add a unregister function in _codecs module #22360

bpo-41842: Add a unregister function in _codecs module #22360

Conversation

shihai1991 commented Sep 22, 2020 • edited Loading

shihai1991 commented Sep 22, 2020 • edited Loading

vstinner left a comment

Choose a reason for hiding this comment

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

vstinner commented Sep 23, 2020

shihai1991 commented Sep 23, 2020

vstinner commented Sep 23, 2020

shihai1991 commented Sep 23, 2020

vstinner left a comment

Choose a reason for hiding this comment

ZackerySpytz Sep 26, 2020

Choose a reason for hiding this comment

vstinner Sep 26, 2020

Choose a reason for hiding this comment

shihai1991 Sep 26, 2020

Choose a reason for hiding this comment

vstinner commented Sep 28, 2020

shihai1991 commented Sep 29, 2020 • edited Loading

shihai1991 commented Sep 22, 2020 •

edited

Loading

shihai1991 commented Sep 22, 2020 •

edited

Loading

shihai1991 commented Sep 29, 2020 •

edited

Loading