Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure private PyPi repositories in the environment #471

21 changes: 18 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,13 +154,28 @@ The default category is `main`.
[environment.yml][envyaml], using a vendored copy of [Poetry's][poetry] dependency solver.

### private pip repositories
Right now `conda-lock` only supports [legacy](https://warehouse.pypa.io/api-reference/legacy.html) pypi repos with basic auth. Most self-hosted repositories like Nexus, Artifactory etc. use this. To use this feature, add your private repo into Poetry's config _including_ the basic auth in the url:
Right now `conda-lock` only supports [legacy](https://warehouse.pypa.io/api-reference/legacy.html) pypi repos with basic auth. Most self-hosted repositories like Nexus, Artifactory etc. use this. You can configure private repositories by setting the `CONDA_LOCK_PYPI_REPOSITORY_<custom-repository>` environment variable to the URL of the private repo, _including_ any basic auth. For example:

```bash
poetry config repositories.foo https://username:password@foo.repo/simple/
export CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE="https://username:password@example.repo/simple"
```

The private repo will be used in addition to `pypi.org`. For projects using `pyproject.toml`, it is possible to [disable `pypi.org` entirely](#disabling-pypiorg).
Alternatively, you can use the `poetry` configuration file format to configure private PyPi repositories. The configuration file should be named `config.toml` and have the following format:

```toml
[repositories.example]
url = "https://username:password@example.repo/simple"
```

The discoverable location of this file depends on the operating system:

- **Linux**: `$XDG_CONFIG_HOME/pypoetry-conda-lock` or `~/.config/pypoetry-conda-lock`
- **OSX**: `~/Library/Application Support/pypoetry-conda-lock`
- **Windows**: Check output of `python -c 'from conda_lock._vendor.poetry.locations import CONFIG_DIR;print(CONFIG_DIR)'`

Comment on lines +163 to +175
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maresb I have restored documentation for the existing repository configuration method here.

Since the previous documentation was incorrect, I have changed the documented process for configuring it this way. This is because the poetry config repositories.foo will write configuration to the wrong appdirs directory for conda-lock to discover (~/.config/pypoetry vs ~/.config/pypoetry-conda-lock).

I have also written it in a way which avoids users needing to install the exact version of poetry vendored by conda-lock.

Also note: I couldn't decipher which directory to recommend for windows, as I suspect its platform-dependent, so that's the best I could do. The old documentation didn't have this problem for the same reason it didn't work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for catching this! Seems like we broke this in #342.


Private repositories will be used in addition to `pypi.org`. For projects using `pyproject.toml`, it is possible to [disable `pypi.org` entirely](#disabling-pypiorg).


### --dev-dependencies/--no-dev-dependencies

Expand Down
15 changes: 15 additions & 0 deletions conda_lock/pypi_solver.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import os
import re
import sys

from functools import lru_cache
from pathlib import Path
from typing import TYPE_CHECKING, Dict, List, Optional
from urllib.parse import urldefrag, urlsplit, urlunsplit
Expand Down Expand Up @@ -373,6 +375,9 @@ def _prepare_repositories_pool(allow_pypi_requests: bool) -> Pool:
factory = Factory()
config = factory.create_config()
repos = [
factory.create_legacy_repository({"name": name, "url": url}, config)
for name, url in _parse_repositories_from_environment().items()
] + [
Comment on lines +378 to +380
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please write a test for _prepare_repositories_pool to exercise this functionality?

factory.create_legacy_repository(
{"name": source[0], "url": source[1]["url"]}, config
)
Comment on lines 381 to 383
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left the previous method here so this is not a breaking change.

However since this is exposing an implementation detail as an interface, it might be worth adding a deprecation warning?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we're ready yet to deprecate it. I'm very eager to ditch anything Poetry-related, but setting up environment variables can be a pain, and until we get something better in place, some people may prefer the Poetry interface.

Expand All @@ -383,6 +388,16 @@ def _prepare_repositories_pool(allow_pypi_requests: bool) -> Pool:
return Pool(repositories=[*repos])


@lru_cache(maxsize=1)
def _parse_repositories_from_environment() -> Dict[str, str]:
env_prefix = "CONDA_LOCK_PYPI_REPOSITORY_"
return {
key[len(env_prefix) :].lower(): value
for key, value in os.environ.items()
if key.startswith(env_prefix)
}


def _strip_auth(url: str) -> str:
"""Strip HTTP Basic authentication from a URL."""
parts = urlsplit(url, allow_fragments=True)
Expand Down
26 changes: 25 additions & 1 deletion tests/test_conda_lock.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,13 @@
)
from conda_lock.models.channel import Channel
from conda_lock.models.lock_spec import VCSDependency, VersionedDependency
from conda_lock.pypi_solver import _strip_auth, parse_pip_requirement, solve_pypi
from conda_lock.pypi_solver import (
_parse_repositories_from_environment,
_prepare_repositories_pool,
_strip_auth,
parse_pip_requirement,
solve_pypi,
)
from conda_lock.src_parser import (
DEFAULT_PLATFORMS,
LockSpecification,
Expand Down Expand Up @@ -1673,6 +1679,24 @@ def test__extract_domain(line: str, stripped: str):
assert _extract_domain(line) == stripped


def test_parse_repositories_from_environment():
# Given a private repository is configured in the environment
try:
_parse_repositories_from_environment.cache_clear()
repository_url = "https://username:password@example.repo/simple"
os.environ["CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE"] = repository_url
# When I prepare the repositories pool
pool = _prepare_repositories_pool(allow_pypi_requests=False)
# Then the private repository is included in the pool
assert pool.repositories, "No repositories were detected"
assert (
pool.repositories[0].url == repository_url
), "Detected repository has incorrect URL"
finally:
_parse_repositories_from_environment.cache_clear()
del os.environ["CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE"]


def _read_file(filepath: "str | Path") -> str:
with open(filepath, mode="r") as file_pointer:
return file_pointer.read()
Expand Down
Loading