allow private flake inputs

[nrdxp]

Is your feature request related to a problem? Please describe.
The current flake inputs spec seems to me to conflate two different concerns.

1. It provides a way for the author to declare a user modifiable (via follows, registry, etc) API for the inputs to the flake
2. It provides a convenient shorthand to the user, from having to constantly update hashes manually

I fear the allure of convenience of the second point is causing noise in the first. For example, there are a lot of projects where I see a flake being imported simply to work with a single output, or even no output at all, but just call import on it to evaluate the Nix expression, or even just pulling a project source repo via flake inputs.

In addition, there may be a situation where the author has not properly consider the implication of a user writing their own follows and doing so may break their flake, or cause a cache miss, etc.

I feel it would be better to provide a clean split here. Inputs that the author genuinely wishes for the user to be able to override should propagate to consuming lock files, these could be considered public flake inputs. Inputs that are not meant to be modified in any way would then be considered private, they would not be overriable or exposed to the consumer, and they would not propagate to their lock. They can, however, still use follows themselves.

Describe the solution you'd like
Inputs could be considered private by default, and we simply place a new attribute public = true for inputs we wish to be public. In addition, for clarity, public inputs should have to be genuine flakes (no flake = false). This could be combined with subflakes to keep things manageable.

Describe alternatives you've considered
I wrote a bit of a hacky solution to this problem in a work PR whereby I keep a flake for private inputs in a subdirectory. I then call that flake with call-flake.nix manually to avoid appending any of these inputs to the top-level lock file.

We have also begun exploring a similar workflow for divnix/std#111

Additional context
#6626 (this will become an issue for large mono-repos, e.g. if we ever flakify nixpkgs)
#6353 (could be considered an alternative implementation for the same idea)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/private-flake-inputs/21277/5

[blaggacao]

Additional Data Point:

Glossary

Flake Span

: the span of the flake; if no subflakes are used, this equals to the span or scope of the repository (i.e. comprises the entire [git] repository).

Input Span

: the theoretical, semantically concise, span of an input.

Assertion

Flake Span != Input Span.

Inputs, by spanning the entire flake, but not necessarily being used across the entire code base are a lossy specification: their semantically correct scope is lost as there is currently no (blessed) way to implicitly or explicitly convey such scope.
In big projects, this lack of scope aggravates the problems mentioned in this issue report.

Conclusion

In light of this data point, the binary distinction between public & private in the design of flakes might eventually need generalization across scopes; or any other appropriate means need to be devised to remediate the current lack of expressiveness when it comes to flake inputs.

[nrdxp]

I would assume proper scoping could be handled by subflakes, hence my comment:

This could be combined with subflakes to keep things manageable.

From what I understand, #6530 should make subflakes usable without the previously awful UX

[rehno-lindeque]

To be honest, I somewhat struggle to see the benefit of this.

Note that I'm not necessarily against making a clean break between flake inputs and non-flake inputs: maybe every package/overlay/etc needs its own lock file or sub-flake or something for its sources. Some kind of major re-architecting in any case.

(I also don't mean to minimize the performance issues mentioned.)

Never-the-less, I think that restricting "follows" in specific contexts is a bit arbitrary and may just end up creating more awful DIY workarounds when people want to override things in unintended ways

In addition, there may be a situation where the author has not properly consider the implication of a user writing their own follows and doing so may break their flake, or cause a cache miss, etc.

Isn't this a bit of a losing battle? Surely there is always going to be a multitude of ways that overriding any input is going to end up breaking an input flake. I don't really see any way to author a flake that guarantees that it will be forward compatible with future versions of any of its inputs.

The only thing I can think of to get anything close to this would be some kind of enforced semantic versioning of flakes based on outputs (similar to what Elm does for its package publishing).

In addition, for clarity, public inputs should have to be genuine flakes (no flake = false).

Taking an example, should I encounter something like this in an input flake:

foo-bar-patch = {
    url = "https://github.com/foo/bar/compare/f36ab16d7abaf7bda5ead040bf7c1897e546b881...a56fa0934c0579539cca236240f26bda880215b8.patch";
    flake = false;
};

I may need to override this patch with a new one in order to avoid breaking the input flake with my other overrides. (Fair enough: I can override the package instead, but the "follows" pattern is pretty convenient given that it is already a flake input.)

[nrdxp]

To be honest, I somewhat struggle to see the benefit of this.

I don't blame you, but my main concern is actually that this is the sort of thing that you don't realize you need until your stuck in a situation with a lock file that is 37,000 lines and takes 5 seconds just to parse (which is never cached, so on every invocation). Granted the performance of the parser is orthogonal, but it's just one example of what happens if we don't take API level concerns more seriously.

Never-the-less, I think that restricting "follows" in specific contexts is a bit arbitrary and may just end up creating more awful DIY workarounds when people want to override things in unintended ways

This isn't the main point of the suggestion, just a natural side-affect of the fact that private inputs would not propogate to the consumer's lock file, so there is nothing to override.

I don't really see any way to author a flake that guarantees that it will be forward compatible with future versions of any of its inputs.

With private inputs, one could make a flake where all inputs are private to enforce proper hashing (in the case of a heavy package build, for example). @gytis-ivaskevicius has suggested something similar before.

Taking an example, should I encounter something like this in an input flake:

This is exactly the kind of thing I'm trying to discourage here. It's no different then having a nix expression in an upstream repo that calls fetchurl. You'll have to override the package in order to change the source. Say this input was marked as private so you couldn't just override it with a follows, you could still override the package at the call site, which may be more verbose, but also semantically clearer.

[gytis-ivaskevicius]

Alternative approaches:
#6780
#6779

[roberth]

An alternate view on the problem is to first solve

• Reuse input lock files #7730

This removes the unnecessary incentive to keep lock files small. (Not just the incentive; they'll actually be smaller)

What then remains of your problem statement is a solution to forbid overriding (follows) on selected inputs, e.g.

inputs.foo.overridable = false;

Attempts to override the input anyway could be detected at lock mutation time, and/or in call-flake.nix.