Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support finding "editable" type annotated packages made editable through direct_url.json #12313

Closed
hashstat opened this issue Mar 8, 2022 · 2 comments · May be fixed by #12315
Closed

Support finding "editable" type annotated packages made editable through direct_url.json #12313

hashstat opened this issue Mar 8, 2022 · 2 comments · May be fixed by #12315

Comments

@hashstat
Copy link

hashstat commented Mar 8, 2022

Feature

Add the ability for mypy to discover the path of "editable" packages installed with direct_url.json files, as described in PEP-610.

Pitch

As the Python ecosystem continues to move toward PEP-517-compliant build systems, the way editable packages are installed is also changing. PEP-660 uses the editables package as an example of how to install editable wheels using import hooks that act as a proxy for editable packages. Editables uses .pth files to load a module finder to proxy the loading of modules from outside of the packages directory without modifying the Python path. Because the path is left unmodified, the methods described in PEP-561, and currently employed by mypy, fail to find these packages. But pip leaves behind a hint about the package path in the direct_url.json file it installs in the .dist-info directory when the -e/--editable flag is used.

The hatchling build backend uses editables to install editable packages. The problem is that mypy cannot determine that such packages are type annotated because of the limitations imposed by its current PEP-561-based discovery methods. Packages that are not stubs cannot be discovered because they aren't named appropriately. And py.typed files cannot be found because the project is proxied by a single python module and is missed by the current mypy search algorithm because the package files are not found on the python path.

Here is an example editable installation that might be installed using pip install -e path/to/package:

venv/lib/python3.9/site-packages/editable_pkg_typed.pth

import _editable_pkg_typed

venv/lib/python3.9/site-packages/_editable_pkg_typed.py

from editables.redirector import RedirectingFinder as F
F.install()
F.map_module('editable_pkg_typed', '/path/to/package/editable_package_typed/__init__.py')

venv/lib/python3.9/site-packages/editable_pkg_typed-0.dist-info/

INSTALLER
LICENSE
METADATA
RECORD
REQUESTED
WHEEL

venv/lib/python3.9/site-packages/editable_pkg_typed-0.dist-info/direct_url.json

{
    "url": "file:///path/to/package",
    "dir_info": {
        "editable": true
    }
}

/path/to/package/

LICENSE
README.md

/path/to/package/pyproject.toml

[build-system]
requires = ['hatchling']
build-backend = 'hatchling.build'

[project]
name = 'editable_pkg_typed'
version = '0'

/path/to/package/editable_pkg_typed/

__init__.py
py.typed

Adding the path from direct_url.json to those discovered in .pth files should be sufficient for mypy to find these editable packages. I will submit a pull request soon with the suggested functionality.

hashstat pushed a commit to hashstat/mypy that referenced this issue Mar 8, 2022
Discover paths to editable packages by looking at direct_url.json files
-- installed by pip when the -e/--editable flag is used -- and add to
the package search paths.
hashstat pushed a commit to hashstat/mypy that referenced this issue Mar 8, 2022
Discover paths to editable packages by looking at direct_url.json files
-- installed by pip when the -e/--editable flag is used -- and add to
the package search paths.
hashstat pushed a commit to hashstat/mypy that referenced this issue Jun 2, 2022
Find paths to editable packages by looking at direct_url.json files --
installed by pip when the -e/--editable flag is used -- and add to the
package search paths.
@abravalheri
Copy link

Adding the path from direct_url.json to those discovered in .pth files should be sufficient for mypy to find these editable packages. I will submit a pull request soon with the suggested functionality.

Hi @hashstat, although this approach might work for some packages, it may not generalise well...

For example, it is a wide spread practice in the community to organise a project in what is called src-layout (as shown in the PyPA tutorial). This means that between the directory shown in direct_url.json and the actual package directory, there might be a src folder, e.g.:

/path/to/project/
├── LICENSE
├── README.md
├── pyproject.toml
└── src
    └── package
        ├── __init__.py
        └── py.typed

(Of course there might be all sorts of creative ways of organizing a project layout, but I think that at least it is important to consider the src-layout, since it has a non-negligible adoption).

@hashstat
Copy link
Author

hashstat commented Aug 8, 2022

it is a wide spread practice in the community to organise a project in what is called src-layout

Hmm. @abravalheri makes a good point.

We could add a search for modules/packages in the src directory. That addition would likely handle 90% of the cases out there. But now we are making assumptions about the layout of project source code that may or may not be correct.

Unfortunately, there doesn't appear to be a simple and reliable method to determine the path of editable packages installed using import hooks. Looks like we might be stuck using MYPYPATH, unless someone has another idea?

@hashstat hashstat closed this as not planned Won't fix, can't repro, duplicate, stale Nov 16, 2022
bpkroth added a commit to microsoft/MLOS that referenced this issue Jul 10, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (#768)
   See Also: 
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
bpkroth added a commit to bpkroth/MLOS that referenced this issue Jul 10, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (microsoft#768)
   See Also:
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
DelphianCalamity pushed a commit to DelphianCalamity/MLOS that referenced this issue Jul 12, 2024
1. Temporarily avoid recent asyncssh version that breaks the tests
2. Workaround pylance issue with pyproject.toml related changes and pip
editable modules install format (microsoft#768)
   See Also: 
   - microsoft/pylance-release#3473
   May also affect `mypy`:
   - python/mypy#16988
   - python/mypy#12313
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants