Cannot load Python modules if they are in a directory named "test"

[jzvikart]

In the following workflow, the step "Run OK" succeeds, but the step "Run Fail" fails.

The only difference is the name of the directory (test vs. test1).

name: Bug

on: push
jobs:
  reproduce:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.11.8"
    - name: Set up
      run: |
        mkdir test
        echo "print('OK')" > test/a.py
        mkdir test1
        echo "print('OK')" > test1/a.py
    - name: Run OK
      run: python -m test1.a
    - name: Run Fail
      run: python -m test.a

As much as I tried, setting PYTHONPATH did not help to fix the last step.

Outputs:

Run python -m test1.a
  python -m test1.a
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.8/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.8/x64/lib
OK

Run python -m test.a
  python -m test.a
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.8/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.8/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.8/x64/lib
/opt/hostedtoolcache/Python/3.11.8/x64/bin/python: No module named test.a
Error: Process completed with exit code 1.

Needless to say, in normal Python installations the problem does not occur and both test and test1 work.

[aparnajyothi-y]

Hello @jzvikart, Thank you for creating this issue and we will look into it :)

[priyagupta108]

Hello @jzvikart 👋 , The issue you're experiencing arises from a naming conflict with Python's standard library. More specifically, test is a package in Python's standard library, and when you run python -m test.a, Python searches for a.py within the standard library's test module, not within your local test directory. On the other hand, test1 isn't a standard library module, so Python treats it as your custom module and finds a within it.

The differences you're noticing between your local Python environment and the GitHub Actions environment are likely due to variations in how the Python interpreter is invoked and the specific environment configurations.
In a normal Python installation running directly on your machine, when you execute a script like python -m test.a, Python adds the current directory (i.e., the directory from where you ran the command) to the start of sys.path. This is why you can import modules from test or test1 without any issues.
However, in the context of GitHub Actions, each run step starts in a new shell, and the Python interpreter may not automatically add the working directory to sys.path. Consequently, when you try to run a module using python -m, Python may not check the current directory unless it's explicitly added to sys.path or the directory is structured as a proper Python package (i.e., including __init__.py).

Here are a couple of potential workarounds:

1. Add an __init__.py file to the test directory. Python then recognizes test as a package and finds the a module within it.

- name: Set up
  run: |
    mkdir test
    echo "print('OK')" > test/a.py
    touch test/__init__.py

If an __init__.py is added, the folder will be recognized as a package; and since its containing folder appears (by default) earlier on sys.path than the standard library folders (deliberately placed at the end, so that this option exists), import string will import that package.

2. Change the working directory before running the module. Modify your workflow as follows:

- name: Run Fail
  working-directory: ./test
  run: python -m a

3. Run the script directly. Instead of using python -m test.a, try python test/a.py.

Please note that these are potential workarounds. The underlying behavior is due to the specific mechanisms of Python's import system and how it interacts with the -m option and the PYTHONPATH. It's generally a good practice to avoid naming your directories and modules the same as Python's standard library modules to prevent such conflicts. You can check the list of Python standard library modules in the Python documentation.

I hope this provides a clearer explanation of the issue. Feel free to reach out if you need any further clarification.

[jzvikart]

@priyagupta108 There is no such Python standard library "test". As I mentioned, in a standard Python installation (not via GH action setup-python) there is no problem having a test directory named "test". In other words, this problem is specific to GH action setup-python.

[priyagupta108]

@jzvikart,
I understand your concern, but test is indeed a part of the Python standard library. It's mainly used for Python's regression suite.
The problem is that you have a directory named test which conflicts with the package of the same name in the standard library test.

The discrepancy you're noticing between standard Python installation and the GitHub Actions environment can be due to a variety of factors, such as differences in the Python interpreter, the way the Python environment is set up, the way the PYTHONPATH is configured, etc.
To illustrate, here's a screenshot from my local setup, which shows the same error:

The initial idea of requiring __init__.py was to prevent directories named like test from shadowing standard modules/packages.
This is a common pitfall, often referred to as The __init__.py trap. You can read more about it here.
The easiest and most robust solution is to avoid using package and module names that conflict with those in the standard library.
I hope this explanation addresses your concern. Please let me know if you have any further questions.

[jzvikart]

Thanks, I know about that package, but as you can read it's for internal use only and not a standard library. However, that's besides the point.

The main point is that setup-python action should provide a Python environment that exactly matches a standard/typical Python installation. Any differences are a potential source of problems. If you are seeing the same error in your local Python environment as you indicated it implies that apparently your local environment is different from my local environment. Would you mind sharing the details about your OS and how you have installed Python? I haven't tested this on other systems yet; if this behaviour is not specific to setup-python (i.e. it appears in other distros) then this ticket is irrelevant.

The solution to rename directory is obvious, but counterintuitive. The name "test" would be quite a common one I imagine. As for __init__.py, please note that I'm using unittest which does its own loading which differs from standard Python module loader.

[priyagupta108]

I agree that test is a common name for directories, especially in projects that include unit tests. The issue here is a consequence of Python's package loading mechanism.
Even though Python's test package is aimed for internal use, its presence can still cause conflicts if a user-defined module or package has the same name.
Before you create any Python module or package, you should make sure there isn't already a module or package by that name. This is a general Python best practice and not specific to GitHub Actions or setup-python.
To check if a module or package exists, you can attempt to import it in the interpreter or use pydoc at the command prompt, like so: python -m pydoc test.
In both GitHub-hosted and self-hosted environments, I've observed this issue to be consistent.
For your reference, here are the details of my local environment:

• Operating System: macOS
• Python Installation Method: Homebrew
• Python Version: 3.12.2

Also, you might find this related Stack Overflow issue helpful: Running unittest with typical test directory structure

[priyagupta108]

Hello @jzvikart ,
I hope the explanations provided throughout this discussion have been helpful. It's always a good practice to ensure that your module or package names do not conflict with existing ones in Python.
This issue is linked with Python's inherent package loading behaviour, which is not specific to GitHub Actions or setup-python. Therefore, I'm proceeding to close this issue. Please feel free to reach out if you have additional questions or concerns. Thank you!