[RFC 0040] "Ret-cont" recursive Nix

rfcs/0000-ret-cont-recursive-nix.md

@@ -0,0 +1,102 @@
+---
+feature: ret-cont-recursive-nix
+start-date: 2019-02-01
+author: John Ericson (@Ericson2314)
+co-authors: (find a buddy later to help our with the RFC)
+related-issues: (will contain links to implementation PRs)
+---
+
+# Summary
+[summary]: #summary
+
+"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
+This avoids some platform-specific contortions relating to nested sandboxing.
+More importantly, it prevents imperative and overly linear direct-style build scripts;
+easy to write but throwing away the benefits of Nix.
+
+# Motivation
+[motivation]: #motivation
+
+The benefits of recursive Nix have been described in many places.
+One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+
+"Ret-cont" recursive nix is short for "return-continuation".
+Consider direct style recursive "nix-build":
+the outer build blocks while the inner one builds, and then continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function passed as an argument, so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
+
+But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
+Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
+Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
+This gives the "return" part of the name.
+
+I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
+This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
+
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
+Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
+With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+
+# Detailed design
+[design]: #detailed-design
+
+Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
+"store" would be a local Nix store limited to just drvs and fixed output builds.
+"drv" would contain a symlink to one of the derivations in the store, the root.
+After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
+\[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
+Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
+\[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+We should muse on this point when implementing to reduce code and perhaps have good insights.]
+
+The *building* itself of derivations is unchanged:
+everything in the previous paragraph just describes marking derivations and post-processing their build results.
+Because drvs and produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate but that's the user's fault.
+We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+\[Instead, the drv->drv substitution would never terminate.]
+Nix can should raise an error rather than looping, but either behavior is permissible.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+ - The opinionated nature may put off those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+
+ - If we ever want full recursive Nix, this doesn't really build in that direction.
+   It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.
+   To me though, this is a feature not a bug; I don't want to go in that direction just yet.
+
+# Alternatives
+[alternatives]: #alternatives
+
+ - Don't allow fixed-output builds.
+   All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
+   But this is much less efficient, and more cumbersome for whatever produces the data.
+
+ - Use a socket to talk to the host daemon.
+   https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
+   This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
+   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
+
+ - Full recursive Nix (builds within builds)
+
+ - Important from derivation.
+   This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+
+ - Keeping the status quo and use vendoring.
+   But then Nix will never scale to bridging the package manager and build system divide.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+
+# Future work
+[future]: #future-work
+
+A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
+This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).