Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest importlib doesn't work with pickle #7859

Closed
yoav-orca opened this issue Oct 5, 2020 · 1 comment · Fixed by #7870
Closed

pytest importlib doesn't work with pickle #7859

yoav-orca opened this issue Oct 5, 2020 · 1 comment · Fixed by #7870
Labels
type: bug problem that needs to be addressed

Comments

@yoav-orca
Copy link

yoav-orca commented Oct 5, 2020

I'm transitioning from pytest regular imports to --import-mode=importlib imports due to namespace collisions in tests. And I'm getting the following crash, no plugins installed, pytest 6.1.1 CPython 3.8.6, the issue reproduces on Mac & Linux.

Create a file named test_xx.py:

import pickle

def do_action():
    pass

def test_x():
    pickle.dumps(do_action)
$ pytest -v --import-mode=importlib test_xx.py
==================================================== test session starts ====================================================
platform darwin -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /Users/yoav/.local/share/virtualenvs/qwer-HyU7-4uu/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/qwer
collected 1 item

test_xx.py::test_x FAILED                                                                                             [100%]

========================================================= FAILURES ==========================================================
__________________________________________________________ test_x ___________________________________________________________

    def test_x():
>       pickle.dumps(do_action)
E       _pickle.PicklingError: Can't pickle <function do_action at 0x10f8aa3a0>: import of module 'test_xx' failed

test_xx.py:7: PicklingError
================================================== short test summary info ==================================================
FAILED test_xx.py::test_x - _pickle.PicklingError: Can't pickle <function do_action at 0x10f8aa3a0>: import of module 'tes...
===================================================== 1 failed in 0.05s =====================================================

Running the same code without pytest or with pytest without importlib work as expected.

pip list:

Package    Version
---------- -------
attrs      20.2.0
iniconfig  1.0.1
packaging  20.4
pip        20.1.1
pluggy     0.13.1
py         1.9.0
pyparsing  2.4.7
pytest     6.1.1
setuptools 50.3.0
six        1.15.0
toml       0.10.1
wheel      0.35.1

I encountered this issue because we have tests using multiprocessing.Pool, and they fail on pickling the target function.

@Zac-HD Zac-HD added the type: bug problem that needs to be addressed label Oct 6, 2020
@aklajnert
Copy link
Contributor

I'd say that it is closely related to #7856. I didn't dig into that yet, but it seems like the problem is with importing, not pickling.

tadeu added a commit to tadeu/pytest that referenced this issue Oct 7, 2020
tadeu added a commit to tadeu/pytest that referenced this issue Oct 7, 2020
tadeu added a commit to tadeu/pytest that referenced this issue Oct 7, 2020
nicoddemus pushed a commit to tadeu/pytest that referenced this issue Apr 2, 2021
nicoddemus pushed a commit that referenced this issue Apr 5, 2021
…sses or pickle (#7870)

Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>

Fixes #7856, fixes #7859
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 14, 2022
…mportlib'

The dummy modules we introduce in `insert_missing_modules` (due to pytest-dev#7856 and pytest-dev#7859)
would cause problems if the dummy modules actually end up replacing modules
which could be imported normally because they are available in `PYTHONPATH`.

Now we attempt to first import the module via normal mechanisms, and only
introduce the dummy modules if the intermediary modules don't actually exist.

Close pytest-dev#9645
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 14, 2022
…mportlib'

The dummy modules we introduce in `insert_missing_modules` (due to pytest-dev#7856 and pytest-dev#7859)
would cause problems if the dummy modules actually end up replacing modules
which could be imported normally because they are available in `PYTHONPATH`.

Now we attempt to first import the module via normal mechanisms, and only
introduce the dummy modules if the intermediary modules don't actually exist.

Close pytest-dev#9645
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug problem that needs to be addressed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants