-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
flake lock parsing performance scaling concerns #6626
Comments
Just took a brief look at the code today to see what can be done. Was hoping there would be some options and/or tunables (e.g. buffer size) in the current library in use but no such luck. In order to improve parsing speed it looks like the only option would be to switch to a different json library. According to synthetic benchmarks (huge grain of salt) we could get the time down by about a factor of 9 or so this way. But as I have interpreted it so far, the library currently in use makes the programmers quality of life much simpler, which should never be discounted either. Another option might be to have some sort of flake lock parse cache, which stores an already parsed lock in a more performance friendly format, and incooperate it into the evaluation cache, but this adds a fairly non-trivial amount of complexity. So all these things considered, and with my own use-case in mind, I still think the best solution is some sort of auto follows as outlined here #5576 and at least partially implemented in #6550. This doesn't account for large monorepos that have a huge flake.lock by necessity, but that is not really the situation I am in personally. Though I am working in a monorepo, a majority of the lock inptus are just replicas of each other, and I don't forsee any major issues with reducing them to a handful or even a single input in most cases.
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/parser-performance-json-vs-nix/20237/4 |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
If we were not to copy everything into the local flake lock unnecessarily, I think we'd solve this performance issue right away. |
Thanks! I will go ahead and close this for now then |
Give https://github.com/fzakaria/nix-auto-follow a try maybe |
Describe the bug
I have a flake.lock at work which has grown to 23742 lines. Early on, I tried to avoid this with explicit
follows
behavior, but because of a tedious UX #5576, coupled with the fact that it doesn't seem possible to follow more than a single level deep (i.e.foo.inputs.bar.inputs.baz.follows
doesn't work) I sort of gave up and just let the thing go for the moment.I started to experience slow evaluations on pretty much everything I called from the flake, and at first I thought maybe there was some IFD somewhere I hadn't noticed, but when I actually went to profile it, I disabled IFD all together and still saw the issue. I began to suspect the lock file when I
strace
d and saw that the majority of the execution time was spent callingbrk()
on a fully eval cached and prebuilt package, because what else could it be doing but reading more data in from the lock if the package and evaluation are already cached?I then removed all outputs of my flake, keeping the inputs and the lock file intact and exported only a single value:
outputs.foo = "bar"
and callednix eval .#foo
, and it took just over 4 seconds. This is a cost that is incurred anytime I reference the flake, since this parsing stage isn't cached in any way. This is particularly annoying when coupled with #3913 which makes just entering a devshell particularly annoying and time consuming, even when it is already built.Expected behavior
Given that we have gone to such great lengths to cache both evaluations and package builds, it would be a shame to have the cli runtime bound solely by the size of the lock file, especially when there is no simple and straightforward way to tell nix something like "make all transitive inputs named
nixpkgs
follow this one" to mitigate it from growing in the first place.Possible solutions
I still think auto follows would be nice, but I could also see legitimate reasons not to use the feature. Ideally we'd want to optimize this parser as much as possible. I'm sure there could be away to avoid calling
brk()
so much. I measured 287 calls on initial analysis.The text was updated successfully, but these errors were encountered: