[RFC 0040] "Ret-cont" recursive Nix

rfcs/0000-ret-cont-recursive-nix.md

@@ -0,0 +1,245 @@
+---
+feature: ret-cont-recursive-nix
+start-date: 2019-02-01
+author: John Ericson (@Ericson2314)
+co-authors: (find a buddy later to help our with the RFC)
+related-issues: (will contain links to implementation PRs)
+---
+
+# Summary
+[summary]: #summary
+
+"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
+This avoids some platform-specific contortions relating to nested sandboxing.
+More importantly, it prevents imperative and overly linear direct-style build scripts;
+easy to write but throwing away the benefits of Nix.
+
+# Motivation
+[motivation]: #motivation
+
+The benefits of recursive Nix have been described in many places.
+One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
+For this case, people usually imagine derivations like
+```nix
+{ stdenv, pkgs, nix }:
+
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
+
+  outputs = [ "out" "dev" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
+```
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+For this, case, the one difference is we need to generate some Nix first.
+```nix
+stdenv.mkDerivation {
+  # ...
+  installPhase = ''
+    bazel2nix # new bit
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
+```
+
+"Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
+The normal variant in the examples above might be termed "direct-style" recursive Nix.
+Consider what happens with the recursive "nix-build" in those examples:
+the outer build blocks while the inner one builds, and then the other one continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function (which is passed as an argument), so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
+But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
+Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
+Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
+This gives the "return" part of the name.
+Both differences together, the first example becomes something like:
+```nix
+{ stdenv, pkgs, nix }:
+
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
+
+  __recursive = true;
+
+  outputs = [ "drv" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
+```
+Note how in this case we don't need to do any "post-processing" of the produced derivation.
+When the outer derivation can just "become" the inner derivation, explicitly copying the derivation outputs like before becomes unnecessary.
+
+So why prefer this variation of the standard design?
+I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
+This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+Building derivations is less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
+
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restrictions that make it a Linux-only feature.
+Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
+With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+Furthermore, we can skip needing to talk to the daemon by just producing a local store:
+```nix
+stdenv.mkDerivation {
+  # ...
+  outputs = [ "drv" "store" ];
+  installPhase = ''
+    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
+```
+This further simplifies the implementation.
+Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.
+
+# Detailed design
+[design]: #detailed-design
+
+Derivations today build outputs, and are associated to those outputs.
+We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
+Derivations that that do so indicate this with some special attribute, say `__recursive`.
+Such derivations must have two outputs, `store` and `drv`.
+`store` would be a local Nix store limited to just drvs and fixed output builds.
+`drv` would contain a symlink to one of the derivations in the store, the root.
+After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
+Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
+
+To faux-formalize everything in the vein of a small-step semantics:
+```
+immediatelyDependsOn(drv0, drv1)
+immediatelyDependsOn(drv1, drv2)
+-------------------------------------------------- deps-trans
+transitivelyDependsOn(drv0, drv2)
+```
+```
+∀<d : transitivelyDependsOn(drv, -)>
+  d ? __recursive == false
+-------------------------------------------------- build-readiness
+isReadyToBuild(drv)
+```
+```
+drv0 : Drv
+∀<o : outputs(drv0)> build_o : StorePath
+∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
+isReadyToBuild(drv0)
+drv0 ? __recursive == false
+-------------------------------------------------- normal-build
+∀<o : outputs(drv0)> assoc(drv0, o, build_o)
+```
+```
+drv0, drv1 : Drv
+drv1path : RelativePath
+∀<o : outputs(drv0)> build0_o : StorePath
+isReadyToBuild(drv0)
+drv0 ? __recursive == true
+drv0.outputs = { "store" = ...; "drv" = ...; }
+build(drv0) = { succeeded = build0; }
+isTrustlessStore(build0_store)
+drv1 = read(build0_store + drv1path)
+readlink(build0_drv) = build0.store + drv1path
+-------------------------------------------------- immediate-drv-deligation
+reducesTo(drv0, drv1)
+```
+```
+drv0, drv1, drv2 : Drv
+reducesTo(drv1, drv2)
+immediatelyDependsOn(drv0, drv1)
+-------------------------------------------------- transitive-drv-deligation
+reducesTo(drv0, drv0[drv2/drv1])
+```
+```
+drv0, drv1 : Drv
+reducesTo(drv0, drv1)
+∀<o : outputs(drv0)> build0_o : StorePath
+∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
+-------------------------------------------------- delegative-build
+∀<o : outputs(drv0)> assoc(drv1, o, build1_o)
+```
+
+## Design Notes
+
+There's a few things we can call out from the faux-formalization.
+
+ - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
+   This is in contrast to normal builds,
+   where the relationship between the derivation and build can only be verified by redoing the build,
+   and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
+
+ - The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+   We should muse on this point, and hopefully write a small-step semantics for both together that is more elegant than the above.
+
+ - The *building* itself of derivations is unchanged.
+   All the magic happens through the `reducesTo` relation.
+
+ - Because drvs can produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate (no `reducesTo` from a derivation to an `isReadyToBuild` derivation) but that's the user's fault.
+   We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+   Nix should raise an error rather than looping, but either behavior is permissible.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+ - The opinionated nature may put off those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+
+ - If we ever want full recursive Nix, this doesn't really build in that direction.
+   It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.
+   To me though, this is a feature not a bug; I don't want to go in that direction just yet.
+
+# Alternatives
+[alternatives]: #alternatives
+
+ - Don't allow fixed-output builds.
+   All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
+   But this is much less efficient, and more cumbersome for whatever produces the data.
+
+ - Use a socket to talk to the host daemon.
+   https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
+   This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
+   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
+
+ - Full recursive Nix (builds within builds)
+
+ - Import from derivation.
+   This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+
+ - Keeping the status quo and use vendoring.
+   But then Nix will never scale to bridging the package manager and build system divide.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+
+# Future work
+[future]: #future-work
+
+A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
+This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).