Add pkgs.callOverlay function

[gytis-ivaskevicius]

Description of changes

Added callOverlay function. This function helps us evaluate overlays without applying them to nixpkgs. Similar idea to extend function but does get polluted with a global context.

Boilerplate for reviewers to test this:

{
  inputs.nixpkgs.url = github:gytis-ivaskevicius/nixpkgs/10e0ac7e0a764e64f967d78869671dda37001160;

  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
    in
    {

      overlays.default = final: prev: {
        a = prev.ruby;
        default = final.callPackage ({a}: a) {};
        inherit (prev) ruby;
      };

      overlays.test = final: prev: {
        a = prev.go;
        c = prev.go;
        d = final.default;
      };

      packages.${system} = pkgs.callOverlays [
        self.overlays.test
        self.overlays.default
      ];

    };
}

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/976

[MagicRB]

This will fail with final.callPackage ive done this in my dotfiles and you need to create a new callPackage with callPackageWith also id like to see a function on lists like callOverlay

[gytis-ivaskevicius]

Ahh, there it is. And here I was thinking that it was too simple. Will fix it later today/tomorrow

[gytis-ivaskevicius]

Ready to be merged

[gytis-ivaskevicius]

I am quite happy with the result. Big thanks to @zimbatm for pointing out that pkgs.extend is not as cheap as I expected.

[roberth]

This is too coupled to stage.nix. It would benefit from being implemented there instead.
With a more "higher order function" flavored solution in stage.nix, we can make this coherent instead of coupled.

[roberth]

What about the other package sets like pkgsStatic or the cross ones?
As a general rule, overriding can not be added externally like you're attempting here. It needs to be integrated in the original structure because of all the possible recursive definitions. In other words, this needs changes in top-level/stage.nix in order to work well.
We haven't even made pkg.override work entirely correctly with cross splicing and overrideAttrs yet, and this change has a lot in common with a new "pkg.override".
For this functionality to not be another series of bugs like we've had with overriding, we need better support for sub-packagesets in stage.nix.
One idea is to make its nixpkgsFun overridable, although that might re-evaluate too much. Another is to put all the extra package sets in their own overlay, so that not just pkgs but all the package sets are easily modifiable.

Here's the general idea. It probably contains some mistakes, but it conveys the idea better than English.

let
  # adaptPackageSet :: attrs -> attrs -> attrs
  # adaptPackageSet = pkgs: definition: ...
  #     Internal to stage.nix.
  #     A function that defines how the package set is extended.
  #     - `pkgs`         an already evaluated (or at least shared) instance of the Nixpkgs fixpoint.
  #     - `definition`   a fresh redefinition of the package set according to the rules of e.g. `pkgsLLVM`, `pkgsCross.<x>`, etc.
  otherPackageSets = adaptPackageSet: self: super: {
    pkgsLLVM = adaptPackageSet super.pkgsLLVM (
      # the existing definition
      nixpkgsFun { /* ... */ }
    );
    # pkgsCross etc
    # pkgs?
  };
  callOverlayOverlay = otherPackageSets (super: definition: callOverlay' super);
  callOverlay' = overlay: callOverlay (/* flip ? */ extends callOverlayOverlay overlay); 
  callOverlay = your definition;
in

# toFix = ... [
# ...
  (otherPackageSets (super: definition: definition))
# ...
  (_: _: { callOverlay = callOverlay'; }) # though not sure if this should be separate (and add a somewhat expensive // on pkgs)
# ...

extend should probably be defined similarly, but ignore the super argument of wrapPackageSet.

I think there's also a possibility of allowing actual overlays to extend the otherPackageSets function, but I'm not sure and that's over-engineering it for the first design step, but it would fix another problem. First priority would be to make something like the above work.

[roberth]

NB this still doesn't cover interactions between extend and callOverlay.

Also how about affixOverlay or attachOverlay? They're not properly merged into self like an overlay normally would, so I think it would be good to add an appropriate verb. extend already doesn't really suggest merging, so I tried to think of something that is even less "merge"-like.

EDIT: add "into self"

[tomberek]

related? https://github.com/nixpkgs-architecture/issues/issues/21

[gytis-ivaskevicius]

Closing due to amount of edge cases and there is no clean way to cover them