Dynamic derivations RFC 92 #4628

Ericson2314 · 2021-03-10T15:24:28Z

See the tests in https://github.com/obsidiansystems/nix/blob/dynamic-drvs/tests/dyn-drv/ for what is being implemented.

Needs:

Make cooler tests
RFC draft
Unstable feature gate to prevent accidental use
Fix Perl bindings

~~Depends on #4594~~
~~Depends on #3959~~
~~Depends on #4543~~
~~Depends on #5364~~
~~Depends on #7601~~
~~Depends on #8353~~
~~Depends on #8369~~
~~Depends on #8813~~
~~Depends on #8829~~
~~Depends on #8927~~
~~Depends on #8938~~

@thufschmitt

These changes were taken from dynamic derivation (NixOS#4628). They somewhat undo the the refactors I first did for floating CA derivations, as the benefit of hindsight + requirements of dynamic derivations made me reconsider some things. They aren't to consequential, but I figured they might be good to land first, before the more profound changes @thufschmitt has in the works.

@thufschmitt

This changes was taken from dynamic derivation (NixOS#4628). It` somewhat undoes the refactors I first did for floating CA derivations, as the benefit of hindsight + requirements of dynamic derivations made me reconsider some things. They aren't to consequential, but I figured they might be good to land first, before the more profound changes @thufschmitt has in the works.

@thufschmitt

This changes was taken from dynamic derivation (NixOS#4628). It` somewhat undoes the refactors I first did for floating CA derivations, as the benefit of hindsight + requirements of dynamic derivations made me reconsider some things. They aren't to consequential, but I figured they might be good to land first, before the more profound changes @thufschmitt has in the works.

@thufschmitt

This changes was taken from dynamic derivation (NixOS#4628). It` somewhat undoes the refactors I first did for floating CA derivations, as the benefit of hindsight + requirements of dynamic derivations made me reconsider some things. They aren't to consequential, but I figured they might be good to land first, before the more profound changes @thufschmitt has in the works.

src/libstore/tests/derivation.cc

doc/manual/src/protocols/derivation-aterm.md

src/libstore/build/derivation-goal.cc

src/libstore/derivations.cc

src/libstore/tests/derivation.cc

- Don't assert: Derivation ATerms are not necessarily produced by Nix, and parsers should always throw graceful errors - Improve error message from `static void except(..)`, shows both what we expected and what we actually got. The intention is that we backport it, and then hopefully a few people might get slightly better errors if they try out new experimental drv files (for RFC 92) with an old version of Nix.

doc/manual/src/protocols/derivation-aterm.md

roberth

Minor suggestions 🚀

src/libcmd/built-path.cc

src/libstore/build/derivation-goal.cc

roberth · 2023-09-07T08:01:42Z

src/libstore/derivations.cc

+    if (std::find_if(
+            inputDrvs.map.begin(),
+            inputDrvs.map.end(),
+            [](auto & kv) { return !kv.second.childMap.empty(); })


Could you add a Derivation method for this?

I made a static function, since nothing else needs it yet. Did you just want it pulled out for legibility, or did you want something publicly in the header in particular?

roberth · 2023-09-07T08:10:43Z

src/libstore/misc.cc

+                    // FIXME: Someday have "deep" realizations analogous to
+                    // deep placeholders, rather than just straight-up
+                    // resolving the computed drv to a store path.


Suggested change

// FIXME: Someday have "deep" realizations analogous to

// deep placeholders, rather than just straight-up

// resolving the computed drv to a store path.

This is not urgent, so best to put this in an issue and reference it from the tracking issue.

Suggested change

// FIXME: Someday have "deep" realizations analogous to

// deep placeholders, rather than just straight-up

// resolving the computed drv to a store path.

// TODO deep resolutions for dynamic derivations, issue #8947, would go here.

I left a reference, just to make this all less cryptic :). Let me know if that is OK.

src/libstore/remote-store.cc

src/libstore/misc.cc

src/libstore/build/derivation-goal.cc

We use the same nested map representation we used for goals, again in order to save space. We might someday want to combine with `inputDrvs`, by doing `V = bool` instead of `V = std::set<OutputName>`, but we are not doing that yet for sake of a smaller diff. The ATerm format for Derivations also needs to be extended, in addition to the in-memory format. To accomodate this, we added a new basic versioning scheme, so old versions of Nix will get nice errors. (And going forward, if the ATerm format changes again the errors will be even better.) `parsedStrings`, an internal function used as part of parsing derivations in A-Term format, used to consume the final `]` but expect the initial `[` to already be consumed. This made for what looked like unbalanced brackets at callsites, which was confusing. Now it consumes both which is hopefully less confusing. As part of testing, we also created a unit test for the A-Term format for regular non-experimental derivations too. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io> Apply suggestions from code review Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

The Derivation parser and old ATerm unfortunately leaves few ways to get nice errors when an old version of Nix encounters a new version of the format. The most likely scenario for this to occur is with a new client making a derivation that the old daemon it is communicating with cannot understand. The extensions we just created for dynamic derivation deps will add a version field, solving the problem going forward, but there is still the issue of what to do about old versions of Nix up to now. The solution here is to carefully catch the bad error from the daemon that is likely to indicate this problem, and add some extra context to it. There is another "Ugly backwards compatibility hack" in `remote-store.cc` that also works by transforming an error. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

dpulls · 2023-09-07T15:07:06Z

🎉 All dependencies have been resolved !

roberth · 2023-09-07T15:36:41Z

That's an MVP for ya!

You'll want to use this with recursive-nix for trying this out with non-trivial builds (writing not a singular drv), but be aware that the interface may change. We want to offer a safer, more limited recursive nix functionality by default, so that a graph of drvs can be constructed, sources can be generated, etc, but that's not been built yet and may or may not use the existing daemon protocol. Again this is experimental. ymmv.

🚀

nixos-discourse · 2023-09-07T17:31:22Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-92-status-update/27441/6

domenkozar · 2023-09-11T17:08:56Z

Is there any documentation how to use dynamic derivations?

Ericson2314 · 2023-09-11T18:35:56Z

Not enough, that is definitely next on the agenda.

Some parts like the outputOf builtin are documented, but other parts are not (like "text" content addressing derivation outputs) because the original concept was already undocumented! And there is definitely no tutorial yet on putting all the pieces together.

For now, I would look at the tests and imitate what they do. Hopefully that is not too cumbersome.

domenkozar · 2023-09-12T08:10:01Z

It seems that dynamic derivations require content addressable store?

Ericson2314 · 2023-09-12T08:58:56Z

Yes. Because derivations themselves are always content addressed, any derivation that produces derivations must be derivation which opts to content-address its outputs.

Ericson2314 force-pushed the dynamic-drvs branch 3 times, most recently from c681bac to 4c74f8f Compare April 7, 2021 01:51

Ericson2314 force-pushed the dynamic-drvs branch 2 times, most recently from 09f5348 to e85248a Compare October 1, 2021 19:11

This comment was marked as resolved.

Sign in to view

Ericson2314 force-pushed the dynamic-drvs branch 3 times, most recently from cfe8286 to 965ad81 Compare October 9, 2021 01:45

Ericson2314 force-pushed the dynamic-drvs branch from 965ad81 to 64fab6e Compare March 11, 2022 01:17

Ericson2314 mentioned this pull request Mar 11, 2022

Overhaul derivation hash modulo somewhat #6229

Merged

This was referenced Mar 21, 2022

Decode string context straight to using StorePaths #6237

Merged

Tracking issue for RFC 92: Dynamic derivations #6316

Open

L-as mentioned this pull request Jun 20, 2022

Granular building of Haskell modules? input-output-hk/haskell.nix#1524

Closed

stale bot added the stale label Sep 21, 2022

Ericson2314 force-pushed the dynamic-drvs branch from 64fab6e to b0b8a91 Compare January 24, 2023 06:26

stale bot removed the stale label Jan 24, 2023

This comment was marked as outdated.

Sign in to view

Ericson2314 force-pushed the dynamic-drvs branch 3 times, most recently from e57531c to f2dc7dc Compare January 24, 2023 21:35

Ericson2314 force-pushed the dynamic-drvs branch from f2dc7dc to 2bc0116 Compare March 1, 2023 04:21

Ericson2314 force-pushed the dynamic-drvs branch from 2bc0116 to 8cbc86e Compare April 2, 2023 18:34

github-actions bot added the with-tests Issues related to testing. PRs with tests have some priority label Apr 2, 2023

Ericson2314 changed the title ~~WIP: Dynamic derivations~~ Dynamic derivations RFC-91 Apr 2, 2023

roberth reviewed Sep 6, 2023

View reviewed changes

Ericson2314 force-pushed the dynamic-drvs branch from 042a74e to c6b9809 Compare September 6, 2023 14:50

Ericson2314 marked this pull request as draft September 6, 2023 14:54

Ericson2314 force-pushed the dynamic-drvs branch from c6b9809 to 7376cad Compare September 6, 2023 14:56

Ericson2314 force-pushed the dynamic-drvs branch from 7376cad to 4fa0603 Compare September 6, 2023 15:49

Ericson2314 marked this pull request as ready for review September 6, 2023 18:19

Ericson2314 force-pushed the dynamic-drvs branch from 4fa0603 to a922a9c Compare September 6, 2023 21:20

roberth reviewed Sep 6, 2023

View reviewed changes

doc/manual/src/protocols/derivation-aterm.md Outdated Show resolved Hide resolved

github-actions bot added the store Issues and pull requests concerning the Nix store label Sep 7, 2023

roberth reviewed Sep 7, 2023

View reviewed changes

Ericson2314 force-pushed the dynamic-drvs branch 3 times, most recently from 7bef7a6 to 95d5ade Compare September 7, 2023 14:37

Ericson2314 and others added 2 commits September 7, 2023 10:39

Ericson2314 force-pushed the dynamic-drvs branch from e2adb14 to 80d7994 Compare September 7, 2023 14:39

Ericson2314 mentioned this pull request Sep 7, 2023

Test derivation parsing failures #8948

Open

roberth approved these changes Sep 7, 2023

View reviewed changes

roberth merged commit e34493a into NixOS:master Sep 7, 2023
8 checks passed

Ericson2314 deleted the dynamic-drvs branch September 7, 2023 16:17

Artturin mentioned this pull request Sep 27, 2023

error: path is not valid using nix ==2.18.0 #9052

Open

Ericson2314 added this to the dynamic-derivations stabilisation milestone Mar 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic derivations RFC 92 #4628

Dynamic derivations RFC 92 #4628

Ericson2314 commented Mar 10, 2021 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

roberth left a comment

roberth Sep 7, 2023

Ericson2314 Sep 7, 2023

roberth Sep 7, 2023

Ericson2314 Sep 7, 2023

Ericson2314 Sep 7, 2023

Ericson2314 Sep 7, 2023

dpulls bot commented Sep 7, 2023

roberth commented Sep 7, 2023

nixos-discourse commented Sep 7, 2023

domenkozar commented Sep 11, 2023

Ericson2314 commented Sep 11, 2023

domenkozar commented Sep 12, 2023

Ericson2314 commented Sep 12, 2023

	// FIXME: Someday have "deep" realizations analogous to
	// deep placeholders, rather than just straight-up
	// resolving the computed drv to a store path.

Dynamic derivations RFC 92 #4628

Dynamic derivations RFC 92 #4628

Conversation

Ericson2314 commented Mar 10, 2021 • edited Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

roberth left a comment

Choose a reason for hiding this comment

roberth Sep 7, 2023

Choose a reason for hiding this comment

Ericson2314 Sep 7, 2023

Choose a reason for hiding this comment

roberth Sep 7, 2023

Choose a reason for hiding this comment

Ericson2314 Sep 7, 2023

Choose a reason for hiding this comment

Ericson2314 Sep 7, 2023

Choose a reason for hiding this comment

Ericson2314 Sep 7, 2023

Choose a reason for hiding this comment

dpulls bot commented Sep 7, 2023

roberth commented Sep 7, 2023

nixos-discourse commented Sep 7, 2023

domenkozar commented Sep 11, 2023

Ericson2314 commented Sep 11, 2023

domenkozar commented Sep 12, 2023

Ericson2314 commented Sep 12, 2023

Ericson2314 commented Mar 10, 2021 •

edited

Loading