nix-build loops while building

[domenkozar]

Nix 2.3.4

Nix seems to loop between the same derivations, see a sample of the log at https://gist.github.com/anamarijas/c5a1ea49ca1e1cbdd3f70f5a8d4aa969

To reproduce:

• git clone https://github.com/cachix/ghcide-nix
• cd ghcide-nix
• cachix use ghcide-nix
• git checkout ghc-8.8
• nix-build

[domenkozar]

I've tested on 2.3.3 and master, same loop.

[andir]

I saw a similar issue a while ago when a binary cache reported a 404 on a file that should be there. Nix did also complain that it didn't know how to build some paths. Can you post the full output that you get from Nix?

[domenkozar]

Full output is 300MB, do you have something to grep for?

I can confirm removing binary caches makes it build.

Was 404 on nar file or narinfo?

[andir]

On 10:42 26.04.20, Domen Kožar wrote: Was 404 on nar file or narinfo?

I honestly do not recall anymore. Could have been either. My guess is on a NAR file but I can't provide evidence. By now I can't reproduce it anymore. I just noticed that the temproots file would eventually fill up the entire disk and only then the build would abort. It was on some cachix bucket shortly after cloudflare was enabled there, if that helps.

[domenkozar]

Thanks @andir I'll take a look at this.

[domenkozar]

My initial hypothesis is that this is an instance of #3373 but during IFD.

[andir]

On 02:39 27.04.20, Domen Kožar wrote: My initial hypothesis is that this is an instance of #3373 but during IFD.

In my case there was no IFD.

[domenkozar]

Refs #1217 #3002

[domenkozar]

I can confirm (even on linux) this is due to missing references from a binary cache, so referenced issues are valid.

I managed to reproduce this again and in the logs it can be observed that Nix is looping between:

some references of path '/nix/store/kk48ba50zs7dj2n30nx4wxxan6273rnq-basement-0.0.11-lib-basement' could not be realised
some references of path '/nix/store/10rgwgpv65fvwajap619pn7i1xj00zky-blaze-builder-0.4.1.0-lib-blaze-builder' could not be         realised

If I try to realize one of those paths manually I get the known error:

$ nix-store -r /nix/store/kk48ba50zs7dj2n30nx4wxxan6273rnq-basement-0.0.11-lib-basement
these paths will be fetched (1.73 MiB download, 17.59 MiB unpacked):
  /nix/store/kk48ba50zs7dj2n30nx4wxxan6273rnq-basement-0.0.11-lib-basement
don't know how to build these paths:
  /nix/store/y62x4jrm775l94gccqkbqfpw2ds2z9bj-basement-0.0.11-lib-basement-doc
error: build of '/nix/store/kk48ba50zs7dj2n30nx4wxxan6273rnq-basement-0.0.11-lib-basement' failed

It seems like currently Nix assumes the binary caches has the full closure, while cachix doesn't enforce this property.

The fact that it loops is probably the result of IFD (maybe enough to trigger this is to import nixpkgs from a pin, etc).

[domenkozar]

@edolstra any idea how easy would it be to fix #3373 and hopefully this?

[domenkozar]

I've addressed this on Cachix to not return narinfo if references are missing. It should still be fixed in Nix I think.

[domenkozar]

This one is particularly interesting:

substitution of '/nix/store/03hgbmq9c6qib7w15f8x1miaciq6bs7x-libyaml-0.1.1.1-lib-libyaml': all references realised
some references of path '/nix/store/03hgbmq9c6qib7w15f8x1miaciq6bs7x-libyaml-0.1.1.1-lib-libyaml' could not be realised

[purefn]

I'm encountering this this week as well. Monday our nix cache, which is hosted on a MinIO server, ran out of space, so I tried purging some old things. I did it the wrong way, looking at narinfo files access times and deleting them, and the nar file they referenced, if they hadn't been accessed in more than 30 days.

What I didn't realize is that multiple narinfo files will reference the same nar file if the output is the same for different derivations. This caused nix cache lookups to find a narinfo file, but fail to find the nar file. The result is the loop mentioned in this issue.

I'm working to track down all the inconsistencies I've created, but it's slow going. I though I had found everything, but am running into another instance and am having a hard time tracking down the culprit.

I would love to see this addressed - I'd even be happy with a hard fail in these situations, with a message saying which substituter and nar file have the inconsistency.

[domenkozar]

We need a small test case to fix this.

I've resolved this issue on Cachix by never serving a narinfo if it doesn't have the whole closure.

[Ericson2314]

This can be closed; github didn't automatically close it for some reason.