Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-Loading the sys or builtins Modules is Problematic #102660

Open
ericsnowcurrently opened this issue Mar 13, 2023 · 2 comments
Open

Re-Loading the sys or builtins Modules is Problematic #102660

ericsnowcurrently opened this issue Mar 13, 2023 · 2 comments
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Mar 13, 2023 •
edited by bedevere-bot
Loading

The sys and builtins modules are special in that they are initialized early in interpreter startup, mostly before the import system. They aren't initialized like other modules. Furthermore, the data they expose is fundamental to the interpreter. Reloading the modules doesn't make much sense.

The simplest thing to do is to disallow manually loading them (e.g. via the loader). At the very least we should handle m_copy for these modules carefully. (This came up while working on gh-101660.)

Linked PRs
@ericsnowcurrently ericsnowcurrently added type-feature A feature request or enhancement interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters 3.12 bugs and security fixes labels Mar 13, 2023
@ericsnowcurrently ericsnowcurrently self-assigned this Mar 13, 2023
@bedevere-bot bedevere-bot mentioned this issue Mar 13, 2023
Merged
ericsnowcurrently added a commit that referenced this issue Mar 14, 2023
@ericsnowcurrently
gh-102660: Handle m_copy Specially for the sys and builtins Modules (g…
cdb21ba 
…h-102661)

It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on gh-101660.)

Note that we might instead end up disallowing re-load for sys/builtins since they are so special.

#102660
@ericsnowcurrently
Copy link
Member Author

I'm leaving this open since we haven't settled whether or not we should disallow directly loading sys/builtins. (That isn't critical to settle in the short-term, so I'm not going to focus much on it for now.)

@ericsnowcurrently ericsnowcurrently removed their assignment Mar 14, 2023
carljm added a commit to carljm/cpython that referenced this issue Mar 14, 2023
@carljm
Merge branch 'main' into inlinecomp2
6c5f269 
* main: (50 commits)
  pythongh-102674: Remove _specialization_stats from Lib/opcode.py (python#102685)
  pythongh-102660: Handle m_copy Specially for the sys and builtins Modules (pythongh-102661)
  pythongh-102354: change python3 to python in docs examples (python#102696)
  pythongh-81057: Add a CI Check for New Unsupported C Global Variables (pythongh-102506)
  pythonGH-94851: check unicode consistency of static strings in debug mode (python#102684)
  pythongh-100315: clarification to `__slots__` docs. (python#102621)
  pythonGH-100227: cleanup initialization of global interned dict (python#102682)
  doc: Remove a duplicate 'versionchanged' in library/asyncio-task (pythongh-102677)
  pythongh-102013: Add PyUnstable_GC_VisitObjects (python#102014)
  pythonGH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (pythonGH-102649)
  pythongh-102627: Replace address pointing toward malicious web page (python#102630)
  pythongh-98831: Use DECREF_INPUTS() more (python#102409)
  pythongh-101659: Avoid Allocation for Shared Exceptions in the _xxsubinterpreters Module (pythongh-102659)
  pythongh-101524: Fix the ChannelID tp_name (pythongh-102655)
  pythongh-102069: Fix `__weakref__` descriptor generation for custom dataclasses (python#102075)
  pythongh-98169 dataclasses.astuple support DefaultDict (python#98170)
  pythongh-102650: Remove duplicate include directives from multiple source files (python#102651)
  pythonGH-100987: Don't cache references to the names and consts array in `_PyEval_EvalFrameDefault`. (python#102640)
  pythongh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct (python#102562)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102631)
  ...
@bedevere-bot bedevere-bot mentioned this issue Mar 16, 2023
Merged
ericsnowcurrently added a commit that referenced this issue Mar 16, 2023
@ericsnowcurrently
gh-102660: Fix Refleaks in import.c (#102744)
2a03ed0 
gh-102661 introduced some leaks. This fixes them.

#102660
carljm added a commit to carljm/cpython that referenced this issue Mar 17, 2023
@carljm
Merge branch 'main' into inlinecomp2
1a8f4a0 
* main: (34 commits)
  pythongh-102701: Fix overflow in dictobject.c (pythonGH-102750)
  pythonGH-78530: add support for generators in `asyncio.wait` (python#102761)
  Increase stack reserve size for Windows debug builds to avoid test crashes (pythonGH-102764)
  pythongh-102755: Add PyErr_DisplayException(exc) (python#102756)
  Fix outdated note about 'int' rounding or truncating (python#102736)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102760)
  pythongh-99726: Improves correctness of stat results for Windows, and uses faster API when available (pythonGH-102149)
  pythongh-102192: remove redundant exception fields from ssl module socket (python#102466)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102743)
  pythongh-102737: Un-ignore ceval.c in the CI globals check (pythongh-102745)
  pythonGH-102748: remove legacy support for generator based coroutines from `asyncio.iscoroutine` (python#102749)
  pythongh-102721: Improve coverage of `_collections_abc._CallableGenericAlias` (python#102722)
  pythonGH-102653: Make recipe docstring show the correct distribution (python#102742)
  Add comments to `{typing,_collections_abc}._type_repr` about each other (python#102752)
  pythongh-102594: PyErr_SetObject adds note to exception raised on normalization error (python#102675)
  pythongh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (python#94468)
  pythonGH-100112:  avoid using iterable coroutines in asyncio internally (python#100128)
  pythongh-102690: Use Edge as fallback in webbrowser instead of IE (python#102691)
  pythongh-102660: Fix Refleaks in import.c (python#102744)
  pythongh-102738: remove from cases generator the code related to register instructions (python#102739)
  ...
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
@ericsnowcurrently @Fidget-Spinner
pythongh-102660: Handle m_copy Specially for the sys and builtins Mod…
f4367d5 
…ules (pythongh-102661)

It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on pythongh-101660.)

Note that we might instead end up disallowing re-load for sys/builtins since they are so special.

python#102660
Fidget-Spinner pushed a commit to Fidget-Spinner/cpython that referenced this issue Mar 27, 2023
@ericsnowcurrently @Fidget-Spinner
pythongh-102660: Fix Refleaks in import.c (python#102744)
9368557 
pythongh-102661 introduced some leaks. This fixes them.

python#102660
@bedevere-bot bedevere-bot mentioned this issue Apr 4, 2023
Merged
ericsnowcurrently added a commit that referenced this issue Apr 4, 2023
@ericsnowcurrently
gh-102660: Fix is_core_module() (gh-103257)
f513d5c 
In gh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on gh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().
gaogaotiantian pushed a commit to gaogaotiantian/cpython that referenced this issue Apr 8, 2023
@ericsnowcurrently @gaogaotiantian
pythongh-102660: Fix is_core_module() (pythongh-103257)
f7e15af 
In pythongh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on pythongh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@ericsnowcurrently @warsaw
pythongh-102660: Handle m_copy Specially for the sys and builtins Mod…
d2aeb2a 
…ules (pythongh-102661)

It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on pythongh-101660.)

Note that we might instead end up disallowing re-load for sys/builtins since they are so special.

python#102660
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@ericsnowcurrently @warsaw
pythongh-102660: Fix Refleaks in import.c (python#102744)
bdc60ae 
pythongh-102661 introduced some leaks. This fixes them.

python#102660
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@ericsnowcurrently @warsaw
pythongh-102660: Fix is_core_module() (pythongh-103257)
2380473 
In pythongh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on pythongh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().
@erlend-aasland erlend-aasland added the 3.13 bugs and security fixes label Jan 5, 2024
@hugovk
Copy link
Member

hugovk commented Mar 15, 2024 •
edited
Loading

Triage: can this be closed or is there more to do?

Edit: I see this was recently move to "Todo" so sounds like there is more. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement
Projects
Subinterpreters
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ericsnowcurrently @hugovk @erlend-aasland