Avoid Building the Entire Project Just to Fetch Dependencies

[nrdxp]

Using sbt-derivation on a work project, my biggest frustration was the fact that I had to essentially always build the project twice, or even three times, to update the hash, to generate the dependency derivation with the proper hash and then finally to build the actual Scala project.

I have been looking for a while for a command that would download all project dependencies without building any artifacts and I seem to have arrived on one that actually works (at least on my work project). If I just add the built in dependency tree plugin to "project/plugins.sbt" with addDependencyTreePlugin, and then run the command sbt dependencyList it pulled all required project dependencies.

There was one remaining issue with the compiler bridge, so the actual Scala package build was still failing with just this command. After trying to find a solution, I discovered after basically testing every possible sbt command in a repl, that the consoleQuick command was the one command I found that initiated a build of the compiler bridge without building the entire rest of the project.

So the final depsWarmupCommand is just sbt "dependencyList ; consoleQuick" <<< ":quit", and with that I was able to generate a working dependency derivation without building any project code. I hope this could be accomplished with sbt-derivation itself (maybe by installing the dependency tree plugin globally for it's call to sbt?) to make for a lighter weight solution than sbt compile by default.

[zaninime]

Hi @nrdxp and thanks for taking the time for looking into this and finding a solution that we could potentially apply directly at the sbt-derivation level.

From your message, I understood that you (more or less) followed these steps:

1. Add the dependency tree plugin (at the project level)
2. Changed the depsWarmupCommand to sbt "dependencyList ; consoleQuick" <<< ":quit"
3. Built the derivation normally

Is that correct? Did your "app" build correctly and run correctly?

Second question: what happens when you consoleQuick without dependencyList before?

[nrdxp]

From your message, I understood that you (more or less) followed these steps...

Yes that is accurate, and the app builds and runs correctly.

Second question: what happens when you consoleQuick without dependencyList before?

With just consuleQuick, not all dependencies were fetched so the build failed.

[zaninime]

@nrdxp, would you mind trying setting this in your project:

{
  # in the derivation
  depsWarmupCommand = ''
    sbt "dependencyPicklePath; compilers"
  '';
}

This makes the trick for me. I will try it on more projects, and I might change it as the new default command for fetching dependencies.

[nrdxp]

So that command still compiles the local project which I was specifically trying to avoid.

[zaninime]

I guess you have multiple modules/projects declared in your build.sbt then. Can you try then externalDependencyClasspath; compilers?

[nrdxp]

That command didn't seem to pull enough dependencies to build Nix with. I am going a bit farther though, and generating a custom lockfile to avoid relying on the FOD all together: #5 (comment)

Maybe it would work with an unmodified sbt-derivation. After reviewing the build error, it looks like it didn't pull the compiler bridge sources. so just modifying it to sbt "externalDependencyClasspath; consoleQuick" <<< ":quit" was enough to rebuild the exact lockfile as my previous command without the need for the dependency tree plugin, so that is definitely an improvement regardless.

That said, on line #29 of my lockfile script I have to manually pull the compiler-bridge sources anyway (for any dependency with compiler-bridge in it's name), since my existing command only pulls them for the version of Scala our project uses but not for the version SBT uses, and it seems both are needed at compile time. So perhaps if I had an alternate way to determine these two version dynamically I wouldn't need the consoleQuick segment either.

As it stands, without consoleQuick the compiler bridge isn't pulled at all for the version of Scala our project is using so that's why my little logic to manually download the source jar for any compiler-bridge entry doesn't trigger.

[tpwrules]

In one of my projects, I ended up putting the following:

  # sbt needs to compile at least one file in order to download all the
  # dependencies, but we don't want it to compile all of our project in order to 
  # save time and resource hassles. so delete the source and compile a fake file
  # to get sbt to do its job properly.
  depsWarmupCommand = ''
    rm -rf src
    mkdir -p src/main/scala
    touch src/main/scala/hi.scala
    sbt compile
  '';

I don't know enough to verify that the reasoning is correct but it seems reasonable enough.