Find a good solution for bloated lock files

[blaggacao]

Out of the frustrations described and linked in : https://discourse.nixos.org/t/private-flake-inputs/21277

• allow a new path schema cell/organelle/flake.nix
• if detected, invoke with the super flake's inputs as overrides via one of the generally available callFlakeWith functions
• otherwise, that flake should only contain organelle specific outputs (-> trivial flakes)

Now, that subflake can specify local ("private") inputs that are of rather scoped character. The toplevel flake's lockfile is kept down, in fact any lockfile is kept down, being nicer to downstream users.

This also may facilitate the self-referential conundrum in mono-repo settings with unbound flake growth.

@nrdxp

[nrdxp]

This sounds like a really good solution for keeping the scope of private flakes in check. Should we have a strict semantic for what can be in the outputs of these organelle local flakes? I would think that we should enforce the outputs = inputs: inputs pattern I used in my bitte PR.

[blaggacao]

Should we have a strict semantic for what can be in the outputs of these organelle local flakes? I would think that we should enforce the outputs = inputs: inputs pattern I used in my bitte PR.

I would look at these as drop-in for default.nix so they should present the same output ideas as their respective default.nix counterparts would.

Instead of:

{inputs, cell}: {}

We may get something like:

{
  inputs = { /* ... */ };
  outputs = inputs: {
    /* ... */
  };
}

I'm currently thinking that in this case, we'd have to forgo the cell & cells inputs, since it wouldn't make sense as a standalone flake in these cases...

But this looks like in need for more thought...

Or: we could assume that the parts that depend on cell or inputs.cells by definition cannot be ripped out of the monorepo (which makes somehow obvious sense).

Another open question would be: how to deal with deSystemize and the instantiated nixpkgs.

[nrdxp]

I was thinking we would just restrict these flakes to re-exporting there inputs so that std could then just deal with them in the same way as it does all other inputs, and any logic must still be contained in the organelle itself

[blaggacao]

That could be a good idea, however, it would o ky cater the locking case, it wouldn't open the door to independently consume that subflake.

But then again, consuming that subflake creates still other issues, which we might not want to incur in a first step.

[blaggacao]

This is another good data point:

• https://github.com/GTrunSec/cells-lab/blob/main/nix/main/library/callFlake.nix#L12

Used here

[nrdxp]

• allow a new path schema cell/organelle/flake.nix

Coming back to this, I think having organelle local flakes is a bit too ganular, after all, it is the cell that is meant to be a logical collection of related code segments, so I'm thinking that private flakes would potentially be better at the cell level.

[blaggacao]

I've recently been wondering if this doesn't introduce prohibitive accidential complexity (as introduced by Fred Brooks' 1987 paper No Silver Bullet) at comparatively little gain?

An optimization that concerns the layout and contents of a machine-written & -read file, might be accidential complexity outside of the actual serializer / deserializer domain. And so are readability aspects for the human eye as a secondary consumer of that file.

On the other hand, unrelated inputs (bloat) that are not adequately abstracted away have a cost and cause accidential complexity all by themselves.

I have a sense that we should wait to see which path upstream takes for the first problem, before we tackle the second.

Or at least, we shouldn't try to solve the first problem, here (serializer/deserializer domain is out of scope for Standard), but, if so, then only the second.

I don't want to be misunderstood: I acknowledge that either way this solves problems that ought to be solved.

[nrdxp]

Well I agree, there is a reason I tried to bring this up upstream first via discourse. Perhaps an actual issue on the Nix issue tracker would be warranted as well. I'm just saying, since we are using flakes in production now, we have a pathway to solve this right now in a way that is well defined.

[blaggacao]

• As far as inefficiencies of the flake lock format concerned, I want to declare them out-of-scope for std. I don't think being a "nix-fixing" layer is a good idea worth persuing.

• As far as additional capabilities a la nvfetcher are concerned, I'd rather see nvfetcher be able to work with flake inputs (instead of reinventing another just as much inefficient lock format).