nixos/*: split docs build #149532

pennae · 2021-12-08T05:56:37Z

Motivation for this change

almost all options documentation is constant, only very few options reference pkgs in attributes that end up in documentation. this fact can be used to build most of options doc in a derivation that can then be cached, and merging the rest of the docs (which can evaluated more cheaply now) separately.

doing so reduces nixos-rebuild time quite significantly (by 20% or more), reducing build time on our system from 9.4 seconds to 7.4 seconds. disabling docs entirely reduces build time to 7.1 seconds, making the docs build as close to free as we can currently get.

Things done

pennae · 2021-12-18T16:34:01Z

most of the modules that have gained an exemption from sandboxed doc builds either interpolate the version of the package they use into doc attributes, or they set relatedPackages. the former isn't easily fixed, the latter could be fixed by separating relatedPackages info from the modules themselves or removing the feature altogether. separating probably doesn't gain us much before there are a lot of users of this feature, but since it's been around for quite a while and hasn't seen much uptake that doesn't seem likely. (and if many users did appear we'd probably lose all the build time improvements the split provices, so maybe it'd be better to remove relatedPackages info as it exists now?)

nixos/modules/misc/documentation.nix

nixos/modules/misc/eval-cacheable-options.nix

nixos/modules/misc/documentation.nix

roberth · 2021-12-18T17:07:09Z

nixos/modules/misc/documentation.nix

-          specialArgs.pkgs = scrubDerivations "pkgs" pkgs;
+        scrubbedEval = evalModules {
+          modules = [ {
+            nixpkgs.localSystem = config.nixpkgs.localSystem;


I don't think you should need this for docs.

Duplicating the evalModules logic is not great for maintainability. Would it be feasible to use extendModules when module filtering is supported by the module system itself?

you're right, that isn't needed here. wonder where that even came from now, probably from attempts to put nixpkgs.nix into the sandbox (which failed anyway) 🤔

re evalModules: we really have to duplicate this because both parts of the build must run with _module.check = false, otherwise the build failed when were testing. not duplicating would be nice, but the savings of this PR come mostly from the docs build not reusing the eval we already have in any way. always open for suggestions though!

You could do essentially extendModules { modules = [ { config._module.args.check = lib.mkForce false; } ]; } for check.
The only challenge is removing some modules.

"some" is a bit of an understatement, there's over 1400 of them at time of writing 😄 filtering loaded modules like that would be possible if that facility existied, but evaluating a few user modules from scratch is probably faster. 🤔

cole-h

Finally making good on my 2-week-old promise...

nixos/lib/make-options-doc/default.nix

nixos/lib/make-options-doc/mergeJSON.py

nixos/modules/config/qt5.nix

nixos/modules/misc/documentation.nix

roberth · 2021-12-18T17:17:54Z

could be fixed by separating relatedPackages info from the modules themselves or removing the feature altogether.

I find it surprising that a bunch of mere strings are causing such an issue. Perhaps the feature could be less strict, though I wouldn't have noticed in practice if you removed it.

pennae · 2021-12-18T17:26:18Z

I find it surprising that a bunch of mere strings are causing such an issue. Perhaps the feature could be less strict, though I wouldn't have noticed in practice if you removed it.

it's not the strings that cause the issue, but how they are handled. the docs mechanism uses those strings as attrnames into pkgs and pulls the package description into the docs.

roberth · 2021-12-18T17:29:52Z

I find it surprising that a bunch of mere strings are causing such an issue. Perhaps the feature could be less strict, though I wouldn't have noticed in practice if you removed it.

it's not the strings that cause the issue, but how they are handled. the docs mechanism uses those strings as attrnames into pkgs and pulls the package description into the docs.

Perhaps these could be replaced by links to search.nixos.org or something. Using the show= query param, example: https://search.nixos.org/packages?show=hercules-ci-agent&sort=relevance&query=hercules-ci-agent

pennae · 2021-12-18T17:49:58Z

turning them into links sounds like a good idea. the manpage build currently renders links in bold red, if we set the link text to something like pkgs.<attribute name> and dropped the package metadata we'd get a nice html manual and the manpage would be sufficient to look up information. does that sound sensible?

roberth · 2021-12-18T17:50:50Z

Yeah, go for it :)

roberth · 2021-12-18T18:36:13Z

nixos/modules/misc/documentation.nix

+    in
+      {
+        lazy = p.right;
+        eager = p.wrong ++ optionals cfg.nixos.includeAllModules (extraModules ++ modules);


For includeAllModules, you'd only have to add the user-provided modules, right? (in addition to p.wrong)

not sure. extraModules was included in the docs build when we started, so we just kept it.

as far as we understand extraModules (via NIXOS_EXTRA_MODULE_PATH) are always loaded but not included by default for performance reasons. it would be nice to include them in the cached part of the build, but like user modules they can reference arbitrary paths so that didn't work out :(

pennae · 2022-01-02T20:01:23Z

anything else we could do to make this any more awesome? :) (would merge it now if we could, two approvals seems good.)

ajs124 · 2022-01-05T15:16:41Z

This (bisect says fc614c3) seems to have broken the installer tests, therefore blocking the nixos channels.

The test fails with:

machine # [    8.959436] dhcpcd[827]: eth0: adding default route via fe80::2
machine # error: path '/nix/store/j02qf8rza5g930jazw4k71g0nigc063f-nixos-22.05pre-git' is not valid
machine # (use '--show-trace' to show detailed location information)
machine: output: 
Test "Perform the installation" failed with error: "command `nixos-install < /dev/null >&2` failed (exit code 1)"

From what I remember the installer tests use closureInfo to populate the store in the vm. So some part of that probably fails now.

vcunat · 2022-01-06T09:50:13Z

So (partial?) revert for now to unblock channels?

pennae · 2022-01-06T09:53:51Z

#153620 has a proposed fix. if that can't be merged quickly let's revert fc614c3 or set documentation.nixos.options.splitBuild = false by default

vcunat · 2022-01-06T13:13:48Z

Unfortunately that still doesn't completely fix the regressions caused by this PR on nixosTests.installer.simple, so channel blocking continues. Some dependencies are now unreachable from within the installer tests, giving lines like:

machine # error: unable to download 'https://salsa.debian.org/debian/keyutils/raw/4cecffcb8e2a2aa4ef41777ed40e4e4bcfb2e5bf/debian/patches/Make-build-reproducible.patch

vcunat · 2022-01-06T13:16:19Z

Usually these were addressed by extending system.extraDependencies in nixos/tests/installer.nix with the right packages (not their build_ dependencies which happen to show up). But I haven't looked into what exactly is newly required here and if it might be an undesirable addition instead.

pennae · 2022-01-06T13:16:48Z

d'oh. didn't let the installer test run through, so we didn't see those. will open a PR to disable the docs split for now until the installer issues are fully sorted.

vcunat · 2022-01-06T13:21:48Z

What I see is certainly just a test-only problem, so perhaps disable in the installer tests only? (unless a good extraDependencies addition in there will occur to us very soon)

vcunat · 2022-01-06T13:24:21Z

What I see failing certainly are dependencies to generate docs (docbook, manual, manpages).

pennae · 2022-01-06T13:25:42Z

we had tried disabling docs in the installer yesterday and had no success. could've been because tiredness, but at the moment i'd rather disable the split build to unblock the channel and have more time to analyse things.

zowoq · 2022-01-08T05:40:42Z

7e28421 broke kubernetes as well.

PR to revert: #153947

zowoq · 2022-01-08T05:45:43Z

And I can't even revert it because it breaks the manual.

pennae · 2022-01-08T05:57:05Z

@zowoq does it unbreak when reverting the kubernetes commits and setting meta.buildDocsInSandbox = false for all kubernetes modules?

pennae · 2022-01-08T06:13:32Z

seems so, the test does complete and the manual builds. #153951

This release fixes the issue (#50) where sandboxed documentation builds [1] fail when using the unstable Nixpkgs channel. [1] NixOS/nixpkgs#149532

bb010g · 2022-06-04T07:40:18Z

This PR broke my NixOS configuration (with modules that customize existing modules) when switching from 21.11 to 22.05, and I had to bisect (git bisect start release-22.05 release-21.11 && git bisect run nix-instantiate nixos -A system) to find this PR.

Here's a sample evaluation failure:

error: The option `security.pam.services.<name>.text' does not exist. Definition values:
       - In `/etc/nixos/modules/modules/pam.nix':
           {
             _type = "override";
             content = ''
               session optional ${pkgs.pam}/lib/security/pam_umask.so 0022
             '';
           ...

       … while evaluating the attribute 'options'

I'm currently trying to figure out how to make this build again without turning nixos.splitOptionDocBuild off. It looks like specialArgs = { options = allOpts; …; } isn't working as intended.

pennae · 2022-06-04T15:12:35Z

can you share a larger example that causes the errors you're seeing? could be that the doc split optimization simply isn't applicable in your case and turning it off could be the only option.

nixos-discourse · 2022-06-18T12:46:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rebuilding-system-using-niv-fails-in-nixos-22-05/19667/7

pennae requested review from adisbladis, dasJ, edolstra, infinisil, nbp, peti and zowoq as code owners December 8, 2021 05:56

github-actions bot added 6.topic: nixos 8.has: documentation 8.has: module (update) labels Dec 8, 2021

pennae requested review from cole-h, jtojnar, roberth and domenkozar December 8, 2021 05:57

pennae marked this pull request as draft December 8, 2021 06:04

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Dec 8, 2021

pennae mentioned this pull request Dec 8, 2021

treewide: more defaultText for options #148785

Merged

13 tasks

adisbladis removed their request for review December 9, 2021 06:40

pennae marked this pull request as ready for review December 18, 2021 13:45

roberth reviewed Dec 18, 2021

View reviewed changes

nixos/modules/misc/documentation.nix Outdated Show resolved Hide resolved

roberth reviewed Dec 18, 2021

View reviewed changes

cole-h reviewed Dec 18, 2021

View reviewed changes

roberth reviewed Dec 18, 2021

View reviewed changes

roberth approved these changes Jan 2, 2022

View reviewed changes

cole-h approved these changes Jan 2, 2022

View reviewed changes

roberth merged commit 70d2730 into NixOS:master Jan 4, 2022

kraem mentioned this pull request Jan 5, 2022

Revert "nixos/documentation: avoid copying nixpkgs subpaths" #153592

Merged

13 tasks

pennae mentioned this pull request Jan 9, 2022

New nixpkgs committers requests #50105

Closed

ncfavier mentioned this pull request Jan 14, 2022

nixos: turn vmWithBootLoader into option (nixos-rebuild build-vm) #151082

Merged

13 tasks

This was referenced Jan 22, 2022

nixos/tests/installer: add bcachefs tests #156071

Merged

nixos/tests/install: Fix after sandboxed-docs change fc614c3 #156213

Merged

zhaofengli added a commit to zhaofengli/colmena that referenced this pull request Jan 26, 2022

Release 0.2.1

52cc4fb

This release fixes the issue (#50) where sandboxed documentation builds [1] fail when using the unstable Nixpkgs channel. [1] NixOS/nixpkgs#149532

erictapen mentioned this pull request Feb 8, 2022

[21.11] Placeholder for meta.buildDocsInSandbox #158642

Merged

13 tasks

infinisil mentioned this pull request Feb 28, 2022

Throw an error for options without a type #162271

Merged

13 tasks

pennae deleted the split-docs-build branch March 24, 2022 08:05

jtojnar mentioned this pull request Jan 9, 2024

Enable split/sandboxed docs build nix-community/home-manager#4875

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/*: split docs build #149532

nixos/*: split docs build #149532

pennae commented Dec 8, 2021 •

edited

Loading

pennae commented Dec 18, 2021

roberth Dec 18, 2021

pennae Dec 18, 2021 •

edited

Loading

roberth Dec 18, 2021

pennae Dec 18, 2021

cole-h left a comment

roberth commented Dec 18, 2021

pennae commented Dec 18, 2021

roberth commented Dec 18, 2021

pennae commented Dec 18, 2021

roberth commented Dec 18, 2021

roberth Dec 18, 2021

pennae Dec 18, 2021

pennae commented Jan 2, 2022

ajs124 commented Jan 5, 2022

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022 •

edited

Loading

vcunat commented Jan 6, 2022

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022

vcunat commented Jan 6, 2022 •

edited

Loading

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022

zowoq commented Jan 8, 2022 •

edited

Loading

zowoq commented Jan 8, 2022

pennae commented Jan 8, 2022

pennae commented Jan 8, 2022

bb010g commented Jun 4, 2022

pennae commented Jun 4, 2022

nixos-discourse commented Jun 18, 2022

nixos/*: split docs build #149532

nixos/*: split docs build #149532

Conversation

pennae commented Dec 8, 2021 • edited Loading

Motivation for this change

Things done

pennae commented Dec 18, 2021

roberth Dec 18, 2021

Choose a reason for hiding this comment

pennae Dec 18, 2021 • edited Loading

Choose a reason for hiding this comment

roberth Dec 18, 2021

Choose a reason for hiding this comment

pennae Dec 18, 2021

Choose a reason for hiding this comment

cole-h left a comment

Choose a reason for hiding this comment

roberth commented Dec 18, 2021

pennae commented Dec 18, 2021

roberth commented Dec 18, 2021

pennae commented Dec 18, 2021

roberth commented Dec 18, 2021

roberth Dec 18, 2021

Choose a reason for hiding this comment

pennae Dec 18, 2021

Choose a reason for hiding this comment

pennae commented Jan 2, 2022

ajs124 commented Jan 5, 2022

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022 • edited Loading

vcunat commented Jan 6, 2022

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022

vcunat commented Jan 6, 2022 • edited Loading

vcunat commented Jan 6, 2022

pennae commented Jan 6, 2022

zowoq commented Jan 8, 2022 • edited Loading

zowoq commented Jan 8, 2022

pennae commented Jan 8, 2022

pennae commented Jan 8, 2022

bb010g commented Jun 4, 2022

pennae commented Jun 4, 2022

nixos-discourse commented Jun 18, 2022

pennae commented Dec 8, 2021 •

edited

Loading

pennae Dec 18, 2021 •

edited

Loading

pennae commented Jan 6, 2022 •

edited

Loading

vcunat commented Jan 6, 2022 •

edited

Loading

zowoq commented Jan 8, 2022 •

edited

Loading