editablePackageSources doesn't work for packages in a src directory

[MartinEekGerhardsen]

Describe the issue

When a package is in a src directory, the editablePackageSources attribute doesn't work.

Additional context

Set up a package app in the main directory, with a single __init__.py file:

def main():
    print("test")

If the flake.nix and pyproject.toml files are set up like this:

default.nix/shell.nix/flake.nix:

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryEnv;

      in
      {
        packages.myapp-env = mkPoetryEnv {
          projectDir = self;
          editablePackageSources.app = ./.;
        };
        devShells.default = pkgs.mkShell {
          inputsFrom = [ self.packages.${system}.myapp-env.env ];
        };
      });
}

pyproject.toml:

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
license = "MIT"
readme = "README.md"

[tool.poetry.scripts]
testing = "app:main"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock:

# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
package = []

[metadata]
lock-version = "2.0"
python-versions = "^3.11"
content-hash = "81b2fa642d7f2d1219cf80112ace12d689d053d81be7f7addb98144d56fc0fb2"

The result is that changing the main function in app/__init__.py will change the behaviour of the testing script (within the devshell defined in the flake).

If this app package is moved to a src folder, and the following changes are applied to the flake.nix and pyproject.toml files, this behaviour is lost, and only recompiling the project will apply the changes. The only changes are changing the editablePackageSources.app attribute from ./. to ./src in the flake.nix file, and adding packages = [{include="app", from="src}] to the [tool.poetry] field of the pyproject.toml file:

default.nix/shell.nix/flake.nix:

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryEnv;

      in
      {
        packages.myapp-env = mkPoetryEnv {
          projectDir = self;
          editablePackageSources.app = ./src;
        };
        devShells.default = pkgs.mkShell {
          inputsFrom = [ self.packages.${system}.myapp-env.env ];
        };
      });
}

pyproject.toml:

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
license = "MIT"
readme = "README.md"
packages = [
    {include = "app", from = "src"},
]

[tool.poetry.scripts]
testing = "app:main"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock:

# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
package = []

[metadata]
lock-version = "2.0"
python-versions = "^3.11"
content-hash = "81b2fa642d7f2d1219cf80112ace12d689d053d81be7f7addb98144d56fc0fb2"

[din14970]

I'm facing the same issue and I have no solution, but I want to at least thank you for sharing a working flake that demonstrates mkPoetryEnv.

[din14970]

I found this in another issue which seems related: #425 . Potentially it's a feature of flakes more so than a bug with poetry2nix?

[MartinEekGerhardsen]

Hmm yeah I think you are right, though still strange that it seems to work better without the src structure 🤷 My main issue here is how I haven't found a good way to automatically recompile the code when I've done some changes, that is often such a struggle haha

[din14970]

I think it's a $PYTHONPATH issue but have to investigate a bit more to be sure. When you run python from your project folder it will be on the python path, and if you have your module at the root it should be directly importable. This is not the case when it is in a src folder. Probably this local folder also takes precedence over the copy of your source files that exist in the nix store, which creates the illusion that your environment is auto updating.

I'm thinking the way to go is

editablePackageSources = {
  myProject = "${builtins.getEnv "PWD"}/src";;
}

as discussed in #425 and https://discourse.nixos.org/t/editable-python-environment-with-poetry2nix/44520

[din14970]

Ok actually I found a few interesting issues and a workaround that seems to work.

Firstly, I've confirmed that without flakes editablePackageSources works as expected. Consider the following shell.nix:

{ pkgs ? import <nixpkgs> {} }:

let
  poetry2nix = pkgs.callPackage (pkgs.fetchgit {
    url = "https://github.com/nix-community/poetry2nix.git";
    rev = "7619e43c2b48c29e24b88a415256f09df96ec276";
    sha256 = "sha256-gD0N0pnKxWJcKtbetlkKOIumS0Zovgxx/nMfOIJIzoI=";
  }) {};

  mkPoetryEnv = poetry2nix.mkPoetryEnv;
in
pkgs.mkShell {
  buildInputs = [
    (mkPoetryEnv {
      projectDir = ./.;
      editablePackageSources = {
        app = ./src;
      };
    })
    pkgs.poetry
  ];
}

If you enter the shell and check the following:

$ nix-shell
$ python
>>> import sys
>>> print(sys.path)
['', '/paths/in/the/nix/store', ..., '/abs/path/to/project/src']
>>> import app
>>> print(app.__file__)
'/abs/path/to/project/src/app/__init__.py'

However, with the equivalent flake

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication;
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryEnv;
      in
      {
        devShells.default = pkgs.mkShell {
          inputsFrom = [ self.devShells.${system}.pythonEnv.env ];
        };

        devShells.pythonEnv = mkPoetryEnv {
          projectDir = ./.;
          editablePackageSources = {
            app = ./src;
          };
        };
      });
}

The behavior is different

$ nix develop
$ python
>>> import sys
>>> print(sys.path)
['', '/paths/in/the/nix/store', ..., '/path/in/the/nix/store/src']
>>> import app
>>> print(app.__file__)
'/path/in/the/nix/store/src/app/__init__.py'

Modifying editablePackageSources to "${builtins.getEnv "PWD"}/src" does not work, in fact it doesn't seem to work at all.

Instead, what does seem to work is using shellHook to manually set the PYTHONPATH:

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {vim 
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication;
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryEnv;
      in
      {
        devShells.default = pkgs.mkShell {
          inputsFrom = [ self.devShells.${system}.pythonEnv.env ];

          shellHook = ''
            export PYTHONPATH=$PWD/src:$PYTHONPATH
          '';
        };

        devShells.pythonEnv = mkPoetryEnv {
          projectDir = ./.;
        };
      });
}

This sort of works although it is not identical to the nix-shell behavior:

$ nix develop
$ python
>>> import sys
>>> print(sys.path)
['', '/abs/path/to/project/src', '/abs/path/to/project', '/paths/in/the/nix/store', ..., ]
>>> import app
>>> print(app.__file__)
'/abs/path/to/project/src/app/__init__.py'

This will only work when you run nix develop inside the root project folder.

I also tried ${builtins.getEnv "PWD"}/src inside shellHook but this did not work because it seems to evaluate to /src.

Likely the reason it works with a flake without src/ is because '' is always on the PYTHONPATH so from the root directory importing your module always works. However this breaks down when you use absolute imports inside your own project.

[MartinEekGerhardsen]

Well done, yeah it seems like you are correct. A bit annoying that this is the case, and at least I can't see any easy way of fixing this, this just seems like python and nix not being friends (as usual).

For the record, one detail which has made it easier for me to work with this is using apps in the flake to reference the scripts defined in the pyproject.toml file, e.g.

...
apps = {
  testing = {
    type = "app";
    program = "${myapp-env}/bin/testing";
  };
};

Then instead of running testing, I'll run nix run .#testing, which makes sure everything is recompiled. Doesn't solve the issue, but makes at least my life easier haha