Suggestion: re-use built dependencies across directories

[mqudsi]

I apologize if this is too brazen a suggestion, but has there been any thought given to having some sort of central cache for each user's builds? I've given this some thought (though I'm sure there's a lot I've missed!) and I think it might be possible to come up with something that allows the safe re-use of previously built dependencies so that one need not re-compile everything for each project.

There are obvious concerns (compiling with different options/flags/parameters, the version being compiled, the compiler version, etc. but these are all things that cargo already does a great job of dealing with and distinguishing when deciding whether or not to rebuild dependencies for a given project.

Then there are issues stemming from potentially building multiple projects at once, directory permissions, etc. that come into play when you talk about having a global cache of anything at all, of course.

This is just one possible approach, but what if upon successfully building library x with settings/release/features/compiler/whatever unique value "abc", a symlink ~/.cargo/cache/x-abc is made pointing to the directory that contains that build? This avoids two different builds trying to build the same project in the same global cache directory at the same time, and any future build can just try accessing that symlink to try and reuse the results of that dependency build. If the directory no longer exists or is no longer matching what it should contain (different "abc" tag, etc.) then that symlink can just be deleted and the dependency recompiled (then a new symlink created).

Again, I know this is really out there and I apologize if I'm overstepping. I just think this could potentially save a good amount of build time and disk space, if done right. However, I'm very aware that this suggestion is fraught with complexities and potential pitfalls, so I'm just bringing this discussion to the table.

[alexcrichton]

This is definitely a possibility! We've had a number of discussions historically and issues and such, but I forget if they're located on the issue tracker or not. In any case one thing you can do today is to try out sccache which does exactly this (caching dependencies when possible). There's some old instructions as well for how to use it.

May be worth giving a shot to see if this has the desired effect on build times?

[luser]

Unfortunately sccache doesn't currently fix this use case due to mozilla/sccache#196, but per the discussion in that bug it seems like it should be straightforward to fix.

[kanekv]

@alexcrichton does cargo always copy files to target folder? In this case extra disk space is used (could symlink instead).

[alexcrichton]

@Kane-Sendgrid Cargo internally prefers hard links wherever it can, only falling back to a copy operation if that fails.

[epage]

This appears to be the same as #5931, closing in favor of that