From 3a8338a6f78ff84b6f652c0784ba593c59a10bcd Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Fri, 1 Feb 2019 01:08:11 -0500
Subject: [PATCH 01/21] Initial draft "ret-cont" recursive Nix

---
 rfcs/0000-ret-cont-recursive-nix.md | 76 +++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 rfcs/0000-ret-cont-recursive-nix.md

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
new file mode 100644
index 000000000..27daf00d7
--- /dev/null
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -0,0 +1,76 @@
+---
+feature: ret-cont-recursive-nix
+start-date: 2019-02-01
+author: John Ericson (@Ericson2314)
+co-authors: (find a buddy later to help our with the RFC)
+related-issues: (will contain links to implementation PRs)
+---
+
+# Summary
+[summary]: #summary
+
+"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
+This avoids some platform-specific relating to nested sandboxing.
+More importantly, it prevents imperative and overly linear direct-style build scripts;
+easy to write but throwing the benefits of Nix.
+
+# Motivation
+[motivation]: #motivation
+
+The benefits of recursive Nix have been described in many places.
+One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+
+"Ret-cont" recursive nix is short for "return-continuation".
+Consider direct style recursive "nix-build":
+the outer build blocks while the inner one builds, and then continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function passed as an argument, so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
+
+But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
+Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
+Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
+This gives the "return" part of the name.
+
+I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
+This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+Building derivations is lot less convenient, but
+
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
+Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
+With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+
+# Detailed design
+[design]: #detailed-design
+
+I defer to https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 with the exact details of setting up the daemon, etc.
+The meaningful distinction of this plan is how the derivation "handoff" works.
+A derivation can call `nix-instantiate` (or really communicate with the daemon however it wants) to create an arbitrary graph of derivations.
+It produces outputs which symlink to those derivations.
+Each of those outputs would then be replaced with the symlink's output of the same name in a downstream derivation.
+\[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+We should muse on this point when implementing to reduce code and perhaps have good insights.]
+Those drvs are then built, perhaps building more derivations like this; it's possible to never terminate but that's the user's fault.
+We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation creating a cycle, we can error out instead of looping.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+We shouldn't be so worried about policing good taste in derivations, and allow full recursive Nix.
+
+# Alternatives
+[alternatives]: #alternatives
+
+Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
+Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I know longer consider the two in conflict.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+
+# Future work
+[future]: #future-work
+
+A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
+This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).

From 3b8422a054dde1ec5a22a80c2a095abf515756db Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Tue, 5 Feb 2019 01:59:28 -0500
Subject: [PATCH 02/21] Fix typos and finish trailing sentance

---
 rfcs/0000-ret-cont-recursive-nix.md | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 27daf00d7..3458e811d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -10,9 +10,9 @@ related-issues: (will contain links to implementation PRs)
 [summary]: #summary
 
 "Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
-This avoids some platform-specific relating to nested sandboxing.
+This avoids some platform-specific contortions relating to nested sandboxing.
 More importantly, it prevents imperative and overly linear direct-style build scripts;
-easy to write but throwing the benefits of Nix.
+easy to write but throwing away the benefits of Nix.
 
 # Motivation
 [motivation]: #motivation
@@ -34,7 +34,8 @@ This gives the "return" part of the name.
 
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
 This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
-Building derivations is lot less convenient, but
+Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 
 Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
 Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
@@ -62,7 +63,7 @@ We shouldn't be so worried about policing good taste in derivations, and allow f
 [alternatives]: #alternatives
 
 Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
-Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I know longer consider the two in conflict.
+Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From 4da91936b0d5746b208f618bf8ad87d3f9c20a8d Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Tue, 5 Feb 2019 02:06:17 -0500
Subject: [PATCH 03/21] Switch to advocating temp store rather than daemon
 socket

Also:

 - Allow fixed output builds (in that temp store)

 - Clean up drawbacks and alternatives.
---
 rfcs/0000-ret-cont-recursive-nix.md | 45 ++++++++++++++++++++++-------
 1 file changed, 35 insertions(+), 10 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 3458e811d..2c1f51bef 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -44,26 +44,51 @@ With "ret-cont" recursive Nix, actual builds are never nested, so we don't need
 # Detailed design
 [design]: #detailed-design
 
-I defer to https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 with the exact details of setting up the daemon, etc.
-The meaningful distinction of this plan is how the derivation "handoff" works.
-A derivation can call `nix-instantiate` (or really communicate with the daemon however it wants) to create an arbitrary graph of derivations.
-It produces outputs which symlink to those derivations.
-Each of those outputs would then be replaced with the symlink's output of the same name in a downstream derivation.
+Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
+"store" would be a local Nix store limited to just drvs and fixed output builds.
+"drv" would contain a symlink to one the derivations in the store, the root.
+After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
+\[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
+Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
 \[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
 We should muse on this point when implementing to reduce code and perhaps have good insights.]
-Those drvs are then built, perhaps building more derivations like this; it's possible to never terminate but that's the user's fault.
-We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation creating a cycle, we can error out instead of looping.
+
+The *building* itself of derivations is unchanged:
+everything in the previous paragraph just describes marking derivations and post-processing their build results.
+Because drvs and produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate but that's the user's fault.
+We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+\[Instead, the drv->drv substitution would never terminate.]
+Nix can should raise an error rather than looping, but either behavior is permissible.
 
 # Drawbacks
 [drawbacks]: #drawbacks
 
-We shouldn't be so worried about policing good taste in derivations, and allow full recursive Nix.
+ - The opinionated nature may put of those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+
+ - If we ever want full recursive Nix, this doesn't really build in that direction.
+   It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.
+   To me though, this is a feature not a bug; I don't want to go in that direction just yet.
 
 # Alternatives
 [alternatives]: #alternatives
 
-Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
-Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+ - Don't allow fixed-output builds.
+   All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
+   But this is much less efficient, and more cumbersome for whatever produces the data.
+
+ - Use a socket to talk to the host daemon.
+   https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
+   This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
+   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
+
+ - Full recursive Nix (builds within builds)
+
+ - Important from derivation.
+   This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+
+ - Keeping the status quo and use vendoring.
+   But then Nix will never scale to bridging the package manager and build system divide.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From 800b5f3a76313aa91ef2b31829bae3c893aae876 Mon Sep 17 00:00:00 2001
From: Langston Barrett <langston.barrett@gmail.com>
Date: Wed, 6 Feb 2019 17:44:10 -0500
Subject: [PATCH 04/21] ret-cont-recursive-nix: Fix typo

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 2c1f51bef..96123b8c3 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -33,7 +33,7 @@ Since the outer derivation evaluates (builds) the inner derivation rather than c
 This gives the "return" part of the name.
 
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
-This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
 Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
 And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 

From f7089838fb498213c7f478b877b9776194d10f90 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= <Mic92@users.noreply.github.com>
Date: Wed, 6 Feb 2019 18:58:01 -0500
Subject: [PATCH 05/21] ret-cont-recursive-nix: Fix typo

Thanks @Mic92

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 96123b8c3..2100cf15b 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -46,7 +46,7 @@ With "ret-cont" recursive Nix, actual builds are never nested, so we don't need
 
 Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
 "store" would be a local Nix store limited to just drvs and fixed output builds.
-"drv" would contain a symlink to one the derivations in the store, the root.
+"drv" would contain a symlink to one of the derivations in the store, the root.
 After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
 \[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
 Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.

From 6a87c1bc5f0d193cb883657d887359e576b598e9 Mon Sep 17 00:00:00 2001
From: Robin Gloster <mail@glob.in>
Date: Thu, 7 Feb 2019 15:48:42 -0500
Subject: [PATCH 06/21] ret-cont-recursive-nix: Fix typo

Thanks @globin

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 2100cf15b..5a7a0292e 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -63,7 +63,7 @@ Nix can should raise an error rather than looping, but either behavior is permis
 # Drawbacks
 [drawbacks]: #drawbacks
 
- - The opinionated nature may put of those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+ - The opinionated nature may put off those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
 
  - If we ever want full recursive Nix, this doesn't really build in that direction.
    It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.

From 36193e5fe9651ab3657935ff2f6a61e80edb99ca Mon Sep 17 00:00:00 2001
From: Langston Barrett <langston.barrett@gmail.com>
Date: Fri, 8 Feb 2019 13:02:55 -0500
Subject: [PATCH 07/21] ret-cont-recursive-nix: Fix typo

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 5a7a0292e..8d505dd81 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -84,7 +84,7 @@ Nix can should raise an error rather than looping, but either behavior is permis
 
  - Full recursive Nix (builds within builds)
 
- - Important from derivation.
+ - Import from derivation.
    This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
  - Keeping the status quo and use vendoring.

From ffb9203496abed07e63b264622c366724c4f5a96 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 12:56:53 -0500
Subject: [PATCH 08/21] ret-cont-recursive-nix: Clean up motivation, adding
 examples

---
 rfcs/0000-ret-cont-recursive-nix.md | 89 ++++++++++++++++++++++++++---
 1 file changed, 81 insertions(+), 8 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 8d505dd81..9d6895fbd 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -19,27 +19,100 @@ easy to write but throwing away the benefits of Nix.
 
 The benefits of recursive Nix have been described in many places.
 One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
-The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+For this case, people usually imagine derivations like
+```nix
+{ stdenv, pkgs, nix }:
 
-"Ret-cont" recursive nix is short for "return-continuation".
-Consider direct style recursive "nix-build":
-the outer build blocks while the inner one builds, and then continues.
-Just as we can CPS-transform programs, reifying the context of a function call as another function passed as an argument, so we can imagine splitting the derivation in two at this blocking point.
-This gives the "continuation" part of the name.
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
 
+  outputs = [ "out" "dev" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
+```
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+For this, case, the one difference is we need to generate some Nix first.
+```nix
+installPhase = ''
+  bazel2nix # new bit
+  for o in $outputs; do
+    pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+    cp -r $pkg ${!o}
+  done
+'';
+```
+
+"Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
+The normal variant in the examples above might be termed "direct-style" recursive Nix.
+Consider what happens with the recursive "nix-build" in those examples:
+the outer build blocks while the inner one builds, and then the other one continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function (which is passed as an argument), so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
 But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
 Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
 Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
 This gives the "return" part of the name.
+Both differences together, the first example becomes something like:
+```nix
+{ stdenv, pkgs, nix }:
+
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
+
+  __recursive = true;
+
+  outputs = [ "drv" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
+```
+Note how in this case we don't need to do any "post-processing" of the produced derivation.
+When the outer derivation can just "become" the inner derivation, explicitly copying the derivation outputs like before becomes unnecessary.
 
+So why prefer this variation of the standard design?
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
 This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
-Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+Building derivations is less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
 And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 
-Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restrictions that make it a Linux-only feature.
 Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
 With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+Furthermore, we can skip needing to talk to the daemon by just producing a local store:
+```nix
+outputs = [ "drv" "store" ];
+installPhase = ''
+  mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+'';
+```
+This further simplifies the implementation.
+Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.
 
 # Detailed design
 [design]: #detailed-design

From 5564fdbd2b24199154f48dc5395e8c1bc32a3bed Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 14:16:08 -0500
Subject: [PATCH 09/21] ret-cont-recursive-nix: Improve syntax highlighting

---
 rfcs/0000-ret-cont-recursive-nix.md | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 9d6895fbd..c958117ba 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -48,13 +48,16 @@ stdenv.mkDerivation {
 The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
 For this, case, the one difference is we need to generate some Nix first.
 ```nix
-installPhase = ''
-  bazel2nix # new bit
-  for o in $outputs; do
-    pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
-    cp -r $pkg ${!o}
-  done
-'';
+stdenv.mkDerivation {
+  # ...
+  installPhase = ''
+    bazel2nix # new bit
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
 ```
 
 "Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
@@ -106,10 +109,13 @@ Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice eithe
 With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
 Furthermore, we can skip needing to talk to the daemon by just producing a local store:
 ```nix
-outputs = [ "drv" "store" ];
-installPhase = ''
-  mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
-'';
+stdenv.mkDerivation {
+  # ...
+  outputs = [ "drv" "store" ];
+  installPhase = ''
+    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
 ```
 This further simplifies the implementation.
 Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.

From 22f8322838a9f7dd0ad364cefc1c3fce0094b3c6 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 19:27:59 -0500
Subject: [PATCH 10/21] Do a lousy job formalizing the detailed design

Break off some previously inline observations into their own subsection.
---
 rfcs/0000-ret-cont-recursive-nix.md | 90 ++++++++++++++++++++++++-----
 1 file changed, 77 insertions(+), 13 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index c958117ba..69231e1cc 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -123,21 +123,85 @@ Derivations remain built exactly as today, with only logic *between* building st
 # Detailed design
 [design]: #detailed-design
 
-Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
-"store" would be a local Nix store limited to just drvs and fixed output builds.
-"drv" would contain a symlink to one of the derivations in the store, the root.
+Derivations today build outputs, and are associated to those outputs.
+We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
+Derivations that that do so indicate this with some special attribute, say `__recursive`.
+Such derivations must have two outputs, `store` and `drv`.
+`store` would be a local Nix store limited to just drvs and fixed output builds.
+`drv` would contain a symlink to one of the derivations in the store, the root.
 After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
-\[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
 Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
-\[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
-We should muse on this point when implementing to reduce code and perhaps have good insights.]
-
-The *building* itself of derivations is unchanged:
-everything in the previous paragraph just describes marking derivations and post-processing their build results.
-Because drvs and produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate but that's the user's fault.
-We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
-\[Instead, the drv->drv substitution would never terminate.]
-Nix can should raise an error rather than looping, but either behavior is permissible.
+
+To faux-formalize everything in the vein of a small-step semantics:
+```
+immediatelyDependsOn(drv0, drv1)
+immediatelyDependsOn(drv1, drv2)
+-------------------------------------------------- deps-trans
+transitivelyDependsOn(drv0, drv2)
+```
+```
+∀<d : transitivelyDependsOn(drv, -)>
+  d ? __recursive == false
+-------------------------------------------------- build-readiness
+isReadyToBuild(drv)
+```
+```
+drv0 : Drv
+∀<o : outputs(drv0)> build_o : StorePath
+∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
+isReadyToBuild(drv0)
+drv0 ? __recursive == false
+-------------------------------------------------- normal-build
+∀<o : outputs(drv0)> assoc(drv0, o, build_o)
+```
+```
+drv0, drv1 : Drv
+drv1path : RelativePath
+∀<o : outputs(drv0)> build0_o : StorePath
+isReadyToBuild(drv0)
+drv0 ? __recursive == true
+drv0.outputs = { "store" = ...; "drv" = ...; }
+build(drv0) = { succeeded = build0; }
+isTrustlessStore(build0_store)
+drv1 = read(build0_store + drv1path)
+readlink(build0_drv) = build0.store + drv1path
+-------------------------------------------------- immediate-drv-deligation
+reducesTo(drv0, drv1)
+```
+```
+drv0, drv1, drv2 : Drv
+reducesTo(drv1, drv2)
+immediatelyDependsOn(drv0, drv1)
+-------------------------------------------------- transitive-drv-deligation
+reducesTo(drv0, drv0[drv2/drv1])
+```
+```
+drv0, drv1 : Drv
+reducesTo(drv0, drv1)
+∀<o : outputs(drv0)> build0_o : StorePath
+∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
+-------------------------------------------------- delegative-build
+∀<o : outputs(drv0)> assoc(drv1, o, build1_o)
+```
+
+## Design Notes
+
+There's a few things we can call out from the faux-formalization.
+
+ - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
+   This is in contrast to normal builds,
+   where the relationship between the derivation and build can only be verified by redoing the build,
+   and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
+
+ - The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+   We should muse on this point, and hopefully write a small-step semantics for both together that is more elegant than the above.
+
+ - The *building* itself of derivations is unchanged.
+   All the magic happens through the `reducesTo` relation.
+
+ - Because drvs can produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate (no `reducesTo` from a derivation to an `isReadyToBuild` derivation) but that's the user's fault.
+   We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+   Nix should raise an error rather than looping, but either behavior is permissible.
 
 # Drawbacks
 [drawbacks]: #drawbacks

From 7f5f85492b7a1326c39b47fd6a3ef23cf9a10b64 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Mon, 11 Feb 2019 11:07:15 -0500
Subject: [PATCH 11/21] ret-cont-recursive-nix: Mention `builtins.exec` in
 alternatives

---
 rfcs/0000-ret-cont-recursive-nix.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 69231e1cc..63fd49fba 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -230,6 +230,11 @@ There's a few things we can call out from the faux-formalization.
  - Import from derivation.
    This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
+ - `builtins.exec` runs an arbitrary command at eval time as the user triggering evaluation.
+   This is highly impure; nothing at all tries to make the environment deterministic.
+   It is useful for writing fetchers that need the impurities to access the internet and secrets, while also managing their own caching.
+   But for everything else, I view it strictly worse than IFD.
+
  - Keeping the status quo and use vendoring.
    But then Nix will never scale to bridging the package manager and build system divide.
 

From 5c9f1fbb1149e9d188c157b905bf3bb563017308 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= <Mic92@users.noreply.github.com>
Date: Mon, 11 Feb 2019 11:11:11 -0500
Subject: [PATCH 12/21] ret-cont-recursive-nix: Fix typo

Thanks @Mic92!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 63fd49fba..cca238d12 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -125,7 +125,7 @@ Derivations remain built exactly as today, with only logic *between* building st
 
 Derivations today build outputs, and are associated to those outputs.
 We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
-Derivations that that do so indicate this with some special attribute, say `__recursive`.
+Derivations that do so indicate this with some special attribute, say `__recursive`.
 Such derivations must have two outputs, `store` and `drv`.
 `store` would be a local Nix store limited to just drvs and fixed output builds.
 `drv` would contain a symlink to one of the derivations in the store, the root.

From 5e56f21125faa6e92ff8269e8fd35655ad9ab958 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 24 Feb 2019 23:57:15 -0500
Subject: [PATCH 13/21] ret-cont-recursive-nix: Remove dangling "$o"

The later examples with `nix-instantiate` automatically get all the outputs of the final rewritten drv, so there's no output iteration needed. `"$o"` was mistakenly copied over from the earlier examples.

Thanks to @ocharles for asking me the question that led me to this. Hopefully this change answers that?
---
 rfcs/0000-ret-cont-recursive-nix.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index cca238d12..059cced12 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -91,7 +91,7 @@ stdenv.mkDerivation {
   doBuild = false;
 
   installPhase = ''
-    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
   '';
 }
 ```
@@ -113,7 +113,7 @@ stdenv.mkDerivation {
   # ...
   outputs = [ "drv" "store" ];
   installPhase = ''
-    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
   '';
 }
 ```

From ba7dcce84a6acc6dad03d502ad95aab9cbc4e20c Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Thu, 15 Aug 2019 09:32:26 -0400
Subject: [PATCH 14/21] Update rfcs/0000-ret-cont-recursive-nix.md
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-Authored-By: Domen Kožar <domen@enlambda.com>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 059cced12..d41d1899f 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -3,6 +3,8 @@ feature: ret-cont-recursive-nix
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: (find a buddy later to help our with the RFC)
+shepherd-leader: Franz Pletz
+shepherd-team: Franz Pletz, Eelco Dolstra, Shea Levy, Daniel Peebles
 related-issues: (will contain links to implementation PRs)
 ---
 

From 8bcb4e68d8a5aa69e6cad6f04eb88527d0c20813 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Sat, 2 Nov 2019 09:56:20 -0400
Subject: [PATCH 15/21] ret-cont-recursive: Fix typo

about -> out
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index d41d1899f..b9fa90a71 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -224,7 +224,7 @@ There's a few things we can call out from the faux-formalization.
  - Use a socket to talk to the host daemon.
    https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
    This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
-   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   No direct access to the host daemon rules out a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
    The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
 
  - Full recursive Nix (builds within builds)

From baae1e6b852a703a05997119acd49343717dac71 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 10:56:09 -0400
Subject: [PATCH 16/21] ret-cont: Add examples and expand future work

---
 rfcs/0000-ret-cont-recursive-nix.md | 149 +++++++++++++++++++++++++++-
 1 file changed, 147 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index b9fa90a71..958058994 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -205,6 +205,97 @@ There's a few things we can call out from the faux-formalization.
    We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
    Nix should raise an error rather than looping, but either behavior is permissible.
 
+# Examples
+[examples]: #examples
+
+As a running example, I'll use @matthewbauer's [Reproducible résumé].
+(Do steal the method; don't steal Matt; he works with me!)
+Here's the original `default.nix`, which uses IFD:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+let
+
+README = stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  buildInputs = [ emacs ];
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    cp resume.nix default.nix
+  '';
+};
+
+in import README {inherit nixpkgs;}
+```
+
+The shortest way to make it instead use "Ret-cont" recursive Nix is this:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+
+in stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  outputs = [ "drv" "store" ];
+  buildInputs = [ emacs nix ];
+  __recursive = true;
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    mv $(nix-instantiate --store $store resume.nix --arg nixpkgs 'import ${nixpkgs.path}') > $drv
+  '';
+}
+```
+
+But note how this means we re-run emacs every time anything in Nixpkgs changes, no good!
+Here's a better version which is more incremental:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+let
+
+# Just like original
+README = stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  buildInputs = [ emacs ];
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    cp resume.nix default.nix
+  '';
+};
+
+in stdenv.mkDerivation {
+  name = "readme-outer";
+  unpackPhase = "true";
+  buildInputs = [ nix ];
+  installPhase = ''
+    mv $(nix-instantiate --store $store ${README} --arg nixpkgs 'import ${nixpkgs.path}') > $drv
+  '';
+}
+```
+
+Now only `readme-outer` is rebuilt when Nixpkgs and Nix changes.
+This may still seem wasteful, but remember we still need to reevaluate whenever those changes.
+Now some of that evaluation work is pushed into the derivations themselves.
+
+This is actually a crucial observation:
+A limit to scaling Nix today is that while our builds are very incremental, evaluation isn't.
+But if we can "push" some of the work of evaluation "deeper" into the derivaiton graph itself, we can be incremental with respect to both.
+This means we are incremental at all levels.
+
 # Drawbacks
 [drawbacks]: #drawbacks
 
@@ -248,5 +339,59 @@ The exact way the outputs refer to the replacement derivations / their outputs i
 # Future work
 [future]: #future-work
 
-A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
-This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).
+1. As the example shows, we can push the work of evaluation into builds.
+   This unlocks lots of future work in Nixpkgs:
+
+   - Leveraging language-specific tools to generate plans Nix builds, rather than reimplementing much of those tools.
+     We do this with IFD today, but both due to Hydra's architecture, and concerns about eval times regardless of Hydra, we don't allow this in Nixpkgs.
+     This is a huge loss as we either do things entirely manually (python) Or vender tons of code (Haskell).
+     We will save valuable human time, and start to bridge the distribution / ops vs developer cultural divide by making it easier to work on your own packages.
+
+   - Simply fetching and importing packages which use Nix for their build system, like Nix itself and hydra, rather than vendoring that build system in.
+     This is an easier special case of the above, where the upstream developer knows and loves Nix, and their package has a Nix-based build system.
+     Flakes are supposed to help with this, as is `builtins.fetchTarball` skirting the IFD prohibition.
+     But, it's better if the hydra evaluator can avoid blocking on the download and/or evaluating the Nix expressions therein.
+     "Ret-cont" recursive Nix would allow this by just putting the "outer" derivation in Nixpkgs.
+
+2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
+   With `--pure` mode, Eelco has also talked about opening the door to caching builds.
+   "Ret-cont" recursive Nix is wonderful foundation for that.
+
+   I hope to at least later proposal automatically converting IFD into "Ret-cont":
+
+   IFD is also slow because the evaluator isn't concurrent and so imported derivations get built one at a time.
+   We can fix this somewhat by modifying the evaluator so that evaluation continues where the *value* of the previously-imported derivation isn't needed.
+   But, we still will inevitably get stuck somewhere shallower in the expression when the value being built is needed.
+
+   With "Ret-cont", we can cleverly avoid needing that value in certain common situations.
+   Quite often, IFD is creating a derivation, so we will have something like:
+
+   ```nix
+   "blah blah blah ${/*some expression ... is stuck because deep inside: */ (import /* ... */) /* ... */} blah blah blah"
+   ```
+
+   Without knowing what the value of that expression is, we may reasonably assume it's coercible to a string.
+   If it isn't, well, evaluation will fail anyways.
+   We can then make a "scratch" derivation, however, that reifies the evaluation of the stuck term, we can splice the scratch derivation's hash instead:
+
+   ```nix
+   "bash balh blash /nix/store/123asdf4sdf1g2dfgh34fg8h7fg69876-i-like-to-procrastinate-and-hope-for-the-best blah blah blah"
+   ```
+
+   Now, evaluation truly isn't stuck at all, and we are as free to continue as if there was no IFD at all!
+   The only failure mode would be if the import was a string but *wasn't* a derivation.
+   But, I imagine we can annotate things such that Nix knows when to speculate like this.
+   (c.f. Compiler hot and cold pragmas.)
+   The author of the Code almost always knows what *type* of thing they are splicing, so I would think we could so annotate quite faithfully.
+   I emphasize "type" because if we ever get a type system, this becomes much easier:
+   Specifying types for imported expressions (along with other explicit signature to guide inference) is wholly sufficient to derive the type of all such splices.
+
+  Another future project would be some speculative strictness to allow one round of evaluation to return *both* a partial build plan and stuck imported derivations.
+  Currently the plan must still be evaluated entirely before any building of "actually needed" derivations, i.e. those which *aren't* imported, begins.
+
+3. More broadly, we can get rid of a special notion of "evaluation" entirely.
+   We can think of evaluation today as basically a special case single layer of dynamism, where the outer work (evaluation) is impure.
+   If all `--pure` and IFD evaluation is done inside Nix builds, then the Nix daemon need not even know about the nix language at all.
+   We can have completely separate tools running inside sandboxes that deal with the Nix expression langauge.
+
+[Reproducible résumé]: https://github.com/matthewbauer/resume

From 9448a2a9330386caee38278b3062f2113b796003 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:23:00 -0400
Subject: [PATCH 17/21] ret-cont: Fix syntax error

No `let`, so don't need `in`.
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 958058994..86ae859cf 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -238,7 +238,7 @@ The shortest way to make it instead use "Ret-cont" recursive Nix is this:
 ```nix
 {nixpkgs ? <nixpkgs>}: with import nixpkgs {};
 
-in stdenv.mkDerivation {
+stdenv.mkDerivation {
   name = "README";
   unpackPhase = "true";
   outputs = [ "drv" "store" ];

From 37a643edb3bd507be7d082fbf22c5160e94e32a2 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:23:36 -0400
Subject: [PATCH 18/21] ret-cont: Mention Ninja's upcomming `dyndep` and C++
 oppertunity

---
 rfcs/0000-ret-cont-recursive-nix.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 86ae859cf..cab2a0c5d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -353,6 +353,12 @@ The exact way the outputs refer to the replacement derivations / their outputs i
      But, it's better if the hydra evaluator can avoid blocking on the download and/or evaluating the Nix expressions therein.
      "Ret-cont" recursive Nix would allow this by just putting the "outer" derivation in Nixpkgs.
 
+   As an example of the former, Ninja (is getting)[Ninja Dyndeps] a very similar notion they call `dyndep`s in their upcoming release.
+   This is needed for C++ and Fortran modules.
+   If CMake and other tools use it as expected, we would need "Ret-cont" to automatically translate their build plans for fine-grained builds of large projects like LLVM or Chromium.
+   Shake, soon Ninja, and eventually [LLBuild], are the only general purpose build systems I know that do or aim to do dynamic dependencies, but none of them sandbox.
+   If we become the only way to both correctly and incrementally build modern C++, that will be a huge opportunity for further growth.
+
 2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
    With `--pure` mode, Eelco has also talked about opening the door to caching builds.
    "Ret-cont" recursive Nix is wonderful foundation for that.
@@ -395,3 +401,7 @@ The exact way the outputs refer to the replacement derivations / their outputs i
    We can have completely separate tools running inside sandboxes that deal with the Nix expression langauge.
 
 [Reproducible résumé]: https://github.com/matthewbauer/resume
+
+[Ninja Dyndeps]: https://github.com/ninja-build/ninja/pull/1521
+
+[LLBuild]: https://github.com/apple/swift-llbuild

From 14b134d683b67e45fc0b0e93ad91689721283ab3 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:24:54 -0400
Subject: [PATCH 19/21] ret-cont: Fix missing explicit `outputs` and
 `__recursive`

This was in the "wrapper" derivation example.
---
 rfcs/0000-ret-cont-recursive-nix.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index cab2a0c5d..6e729603c 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -242,8 +242,8 @@ stdenv.mkDerivation {
   name = "README";
   unpackPhase = "true";
   outputs = [ "drv" "store" ];
-  buildInputs = [ emacs nix ];
   __recursive = true;
+  buildInputs = [ emacs nix ];
   installPhase = ''
     mkdir -p $out
     cd $out
@@ -281,6 +281,8 @@ in stdenv.mkDerivation {
   name = "readme-outer";
   unpackPhase = "true";
   buildInputs = [ nix ];
+  outputs = [ "drv" "store" ];
+  __recursive = true;
   installPhase = ''
     mv $(nix-instantiate --store $store ${README} --arg nixpkgs 'import ${nixpkgs.path}') > $drv
   '';

From 1b0a6a1979d13179594e6422ad0a631a0179e4b8 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 5 Nov 2019 13:14:45 -0500
Subject: [PATCH 20/21] ret-cont: "caching builds" -> "caching evaluation"

We already cache builds just fine, thanks!

Thanks @globin for catching
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 6e729603c..fc0b7099d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -362,7 +362,7 @@ The exact way the outputs refer to the replacement derivations / their outputs i
    If we become the only way to both correctly and incrementally build modern C++, that will be a huge opportunity for further growth.
 
 2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
-   With `--pure` mode, Eelco has also talked about opening the door to caching builds.
+   With `--pure` mode, Eelco has also talked about opening the door to caching evaluation.
    "Ret-cont" recursive Nix is wonderful foundation for that.
 
    I hope to at least later proposal automatically converting IFD into "Ret-cont":

From 3fe1c3df6b69a3c9446efc495930f7e5f1714db5 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Thu, 12 Dec 2019 12:46:57 -0500
Subject: [PATCH 21/21] ret-cont: Improve formalism and reference #62

---
 rfcs/0000-ret-cont-recursive-nix.md | 42 ++++++++++++++++++-----------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index fc0b7099d..1b8043210 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -136,14 +136,16 @@ Finally, any uses of the original derivation can be substituted to instead use t
 
 To faux-formalize everything in the vein of a small-step semantics:
 ```
-immediatelyDependsOn(drv0, drv1)
-immediatelyDependsOn(drv1, drv2)
--------------------------------------------------- deps-trans
-transitivelyDependsOn(drv0, drv2)
+∀<d : immediatelyDependsOn(drv, -)>
+  isValue(d)
+drv ? __recursive == false
+drv ? outputHash == false
+-------------------------------------------------- normalized
+isValue(drv)
 ```
 ```
-∀<d : transitivelyDependsOn(drv, -)>
-  d ? __recursive == false
+∀<d : immediatelyDependsOn(drv, -)>
+  isValue(d)
 -------------------------------------------------- build-readiness
 isReadyToBuild(drv)
 ```
@@ -152,29 +154,34 @@ drv0 : Drv
 ∀<o : outputs(drv0)> build_o : StorePath
 ∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
 isReadyToBuild(drv0)
-drv0 ? __recursive == false
--------------------------------------------------- normal-build
+-------------------------------------------------- simple-build
 ∀<o : outputs(drv0)> assoc(drv0, o, build_o)
 ```
 ```
+drv : Drv
+drv.outputs = { "out" = ...; }
+drv ? outputHash == true
+-------------------------------------------------- immediate-reduction-fixed-output
+reducesTo(drv0, stubDrv(drv.outputHash))
+```
+```
 drv0, drv1 : Drv
 drv1path : RelativePath
+drv0.outputs = { "store" = ...; "drv" = ...; }
 ∀<o : outputs(drv0)> build0_o : StorePath
-isReadyToBuild(drv0)
+∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
 drv0 ? __recursive == true
-drv0.outputs = { "store" = ...; "drv" = ...; }
-build(drv0) = { succeeded = build0; }
 isTrustlessStore(build0_store)
 drv1 = read(build0_store + drv1path)
 readlink(build0_drv) = build0.store + drv1path
--------------------------------------------------- immediate-drv-deligation
+-------------------------------------------------- immediate-reduction-drv-deligation
 reducesTo(drv0, drv1)
 ```
 ```
 drv0, drv1, drv2 : Drv
 reducesTo(drv1, drv2)
 immediatelyDependsOn(drv0, drv1)
--------------------------------------------------- transitive-drv-deligation
+-------------------------------------------------- transitive-reduction
 reducesTo(drv0, drv0[drv2/drv1])
 ```
 ```
@@ -190,13 +197,18 @@ reducesTo(drv0, drv1)
 
 There's a few things we can call out from the faux-formalization.
 
+ - When we build a derivation, we don't want it's associated build.
+   This is because it may not exist if it is not in normal form, and may be an intermediate result if the derivation is recursive.
+   Instead, we we want the associated build of the *evaluation* of the given derivation.
+
  - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
    This is in contrast to normal builds,
    where the relationship between the derivation and build can only be verified by redoing the build,
    and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
 
- - The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
-   We should muse on this point, and hopefully write a small-step semantics for both together that is more elegant than the above.
+ - The substitution of drvs in a downstream derivation is very realted to the substitution of drvs for content hashes with the intensional store.
+   I included the normalization of content adressible stores, today done with `hashDerivationModulo` to to help make that clear, and show that we already normalize derivations.
+   I hope to eventually share a formalism for this and RFC #62.
 
  - The *building* itself of derivations is unchanged.
    All the magic happens through the `reducesTo` relation.