Cargo: produce deterministic filenames for build --test and test --no-run

[bruno-medeiros]

I'm doing IDE integration, and I want my IDE to be able to debug the test binaries - for which the IDE has to launch those programs itself, it shouldn't use cargo test. However (unlike --binfor example) neither build --test nor test --no-run produce a deterministic filename, but something random like test1-748b2ca97d589628.exe.

[alexcrichton]

The problem Cargo has to deal with here is that you can test many targets which are all named the same. For example you could have a binary called foo, a library called foo, and an integration test also called foo. All of these cases need to generate binaries with unique names so they can coexist.

I do agree though it's annoying to have to basically do a diff of the output directory before and after a test run to see what binaries were generated.

[bruno-medeiros]

Why not just put the test binaries in their own directory then, like target/debug/tests/ ?
Or prefix the executable name with test. so that integration test foo would generate target/debug/test.foo.exe ? (the target name can't have a dots, so there can't be a conflict if test. is used as file prefix)

[alexcrichton]

The problem is that you can have multiple binaries called foo. We'd need prefixes like test.bin, test.lib, test.test, etc.

[bruno-medeiros]

The problem is that you can have multiple binaries called foo. We'd need prefixes like test.bin, test.lib, test.test, etc.

Which is a perfectly fine solution, no?
Or alternatively, this minor variant of the above:
test.bin.<binName>, test.lib.<libName>, test.<testName>
which is less verbose for the names of the binaries of the test targets.

[bruno-medeiros]

Yet another solution would be to have the Cargo build targets all share the same namespace. Seems like a fine solution if Cargo was just starting out, but at this point it would be a breaking change - so not ideal, I reckon...

[alexcrichton]

There may be a more ergonomic possibility than test.$type.$name, but overall I think it would work out in terms of disambiguation at least.

[bruno-medeiros]

There may be a more ergonomic possibility than test.$type.$name, but overall I think it would work out in terms of disambiguation at least.

Then how about just test.$testname, and make bin and lib a reserved/invalid test name for custom Cargo test targets?

[alexcrichton]

Hm actually now that I think about it we probably want to continue the pattern of <name>-* so previous scripts which use that kind of glob today will continue to work, and that means the names could be something like <name>-test.<kind>. I think it's fine to leave the <kind> in there as reserving names may be a bit late in the game at this point.

[bruno-medeiros]

I think it's fine to leave the in there as reserving names may be a bit late in the game at this point.

Well, Rust may be 1.0, but Cargo is only version 0.2, so it can be argued that a minor breaking change would be acceptable. But I don't know how strict your policy aims to be with regards to breaking changes in Cargo.
(personally, I'm not that fussed about it either way, as long as it becomes a deterministic naming scheme)

[alexcrichton]

Despite Cargo's version it actually needs to be quite stable today, so I'd prefer to keep at least roughly the same pattern that we have today.

[FlorentBecker]

How about (additionnally?) making cargo run --no-test output the name of the executables it builds on stdout? This way, cargo test --no-run -q gives the path to the executable (rebuilding it if needed).

[alexcrichton]

@FlorentBecker my preference here would be to just have deterministic filenames for now, but that's a possible alternative if it doesn't work out!

[adrianheine]

I'd have to say that I would prefer to have a way to get the file name from cargo instead of being able to guess it myself.

[posborne]

I have to agree with @adrianheine and @FlorentBecker. It would be very nice to have cargo give the output of test discovery. To make the output parseable in scripts and the like, I would propose that something like a cargo test --print or cargo test --discover be added that simply prints the relative path to each test executable. This option could be used along with others to limit the scope of the discovery.

Currently, I am working on adding infrastructure to nix based on libc. Unlike libc, nix does not currently have most of its tests built separately in a single executable. Right now, after cross compiling the executables there is no great way to get the paths to the test executables and pass execute them in qemu. I can glob but even then it involves a repetition of the individual tests that are in the root Cargo.toml.

[bruno-medeiros]

@alexcrichton If compatibility with existing build scripts is important (implying the continuing of the pattern of <name>-*) , why not simply generate the deterministic filenames when an additional flag (--simple-filenames?) is passed to cargo test? This way existing build scripts work the same, and only new tools use the additional flag.

@posborne 's additional suggestion of enabling Cargo to output the filenames is good as well. It enables tools to figure out the executables without having to be able to parse Cargo.toml and figure out the Cargo targets beforehand. But this suggestion should be an addition, not the main solution, there should still be a way to generate static/deterministic filenames.

[alexcrichton]

Yeah at this point I think that we should probably just invent some scheme that has predictable names but lacks hashes. Along those lines I'd be fine with basically anything that kept the convention of <name>-* and then the * was just replaced by something deterministic.

[bruno-medeiros]

Hum, so like <name>-test for integration-style tests, <name>-test.lib for lib tests and <name>-test.bin for bin tests?

[alexcrichton]

That sounds pretty reasonable to me, yeah!

[jan-hudec]

<name>-test.lib, <name>-test.bin

I would perhaps avoid the ., which might be somewhat confusing on Windows, and suggest <name>-lib-test and <name>-bin-test. With -test always at end—or beginning—so that *-test (test-*) matches all test binaries.

[alexcrichton]

Unfortunately using -test as a suffix or prefix can conflict with other targets. For example if you have binaries called test-foo-lib and foo-lib-test, then we can't generate a binary for a corresponding library called foo (because - is valid in crate names). What's the confusion on Windows you're thinking of though?

[jan-hudec]

Then <name>-test for the tests under tests/ wouldn't work either, right?

The confusion under Windows I feared was that Windows by default hide the extension in various places and don't do it in a particularly consistent way. And because .bin and .lib are commonly used as suffixes, sometimes you might be looking at foo-test.lib.exe and confuse it for foo-test.lib or vice versa.

If you want to have . in the names to avoid conflict, I would then suggest: test.name(.exe), test.bin.name(.exe) and test.lib.name(.exe). These would keep the advantage that test.* is all tests, the . should still prevent conflicts and the name at the end would be unlikely to resemble a real extension.

[alexcrichton]

Hm this actually gives me an idea! So I'd like to keep the same names we have today in case any scripts are relying on them, but we could perhaps do something like:

1. Generate all executables into a tests/ directory with deterministic names
2. Hard-link all those executables up one level to where they are today with today's names

That way we could slowly phase out the old locations and we could continue to use the new ones today!

[bruno-medeiros]

The confusion under Windows I feared was that Windows by default hide the extension in various places and don't do it in a particularly consistent way.

If that's confusing, Windows users should disable the Windows Explorer option that hides file extensions. I suspect most Windows developers and power users do that already anyways (I certainly do). And IDEs certainly don't hide extensions.

[bruno-medeiros]

@alexcrichton Yeeeesh, hard links... you sure that's a good idea? Sounds like a really heavy handed solution. Do all the OSes that Rust targets have filesystems that support hard links? What if one wants to use a filesystem that doesn't support hard links? Imagine for example someone has a Rust project on a portable FAT32 USB stick or something like that. Or a network share? It's a really odd and rare scenario, but I think it might realistically happen.
What library would you even use to create hard links in a portable way, does such a thing exist in Rust?

[kamalmarhubi]

Re library to use, std::fs::hard_link() exists :-)

[bruno-medeiros]

@kamalmarhubi Oh cool, didn't know about that.
I still think it's not a clean solution though, it's better to fix this in a way that works independently of the underlying file system.

[kamalmarhubi]

What if one wants to use a filesystem that doesn't support hard links? Imagine for example someone has a Rust project on a portable FAT32 USB stick or something like that.

I haven't thought about if hardlinks are the right approach, but this is a real concern. I think between FS support and confusing Explorer defaults, I'd prefer the confusion over reduced FS support.

[andrewdavidmackenzie]

I am looking for something like this, so on macos I can code-sign the test binaries created (to avoid the network access permission modal dialog that stops my build) before they are run (from a Makefile).

But it looks like this conversation petered out and wasn't concluded in 2016-2018!!

I'd like:

• build test binaries (cargo test --no-run?)
• do my code signing
• run tests

[jschwe]

@andrewdavidmackenzie For your specific problem you could probably use CARGO_TARGET_<triple>_RUNNER and set it to a simple script, which

• first does code signing
• then runs the test

The path to the test exectuable is passed as an argument to the runner script.

[andrewdavidmackenzie]

Thanks for the suggestion. Trying it now.

It looks like I can't set in my workspace Cargo.toml and will need to add to each member Cargo.toml?

I'm struggling with the format:

[target.x86_64-apple-darwin]
runner = "../mac_test_runner.sh"

causes

warning: /Users/andrew/workspace/flow/flowr/Cargo.toml: unused manifest key: target.x86_64-apple-darwin.runner

[jschwe]

@andrewdavidmackenzie this configuration option needs to be set in a .cargo/config.toml, not in a Cargo.toml. you can place this configuration file in your workspace root. (The .cargo folder must be a parent directory of the folder cargo was invoked from.)

[andrewdavidmackenzie]

Got it working - thanks for your help!

[amyspark]

Hi all, adding this would be great for Meson users. The tests' building step could be decoupled and kept away from ninja test in this way, through the usage of custom_target's outputs.

[Kekun]

Same here, I spent quite some time figuring out how to generate Meson tests from the produced test binaries, but it's really dirty and frankly unreliable. It's needed so I can make my Cargo-powered tests live with all the other tests of my project, like my app's metadata checkers, and so they can all integrate well with IDE and my CI.