Make it possible to run binaries produced by cargo directly.

[matklad]

EDIT: 90% of this is solved by --message-format=json flag.

Hi! This is a followup of #1924 and #1726.

Problem

Sometimes one needs to run an executable produced by Cargo (be it example, binary or doc/unit/integration test) directly.

The most common use case is debugger integration in the IDE, but other examples include strace, perfrecord and similar tools.

Peculiarities

cargo test can produce more than one binary. We need to handle this somehow.

Both cargo run and cargo test have different flavors for specifying profiles, features, targets and arguments for the binary. Ideally, it should be easy to learn the name of output file by the cargo command.

Cargo can setup LD_LIBRARY_PATH when running a binary, it would be good to be able replicate this as well.

Solutions

--with wrapper

Add a --with option to the run family of commands to allow to specify a custom wrapper. There's a stale pull request implementation in #1763.

This would probably be the most useful option on the command line. However, it is not flexible enough: there's still an intermediate Cargo process around, and this won't be enough for IntellJ Rust: we launch debugger process first and then send it a command to launch the debugee.

Also, this can be implemented as an external subcommand on top of other options, so that you can run
cargo with-wrapper strace run --release --bin foo.

--output-file option

We can implement #1706, than clients could specify the name they want. This is a nice option, but it's not perfect: the binary may assume certain location and use something like ../../foo/bar to get resources at run time or something like that. While arguably you should not do this, ideally we should be able able to run binaries exactly as Cargo does. Also with explicit output file the client has to manage temporary directories, which is some extra work.

Stable file names

We can make the names of binaries predictable. This is perhaps the simplest option, but I don't like it for various reasons:

• We already have debug and release profiles, which affect the output path. This means that clients should implement some logic to get the current profile, which may be brittle.

• This requires stabilizing target directory layout and may make future features harder. What if one day we would like to add custom profiles and an ability to configure the custom profile via environment variables/config files?

• This does not allow to easily derive the binary name given the cargo command, which again means some logic on the client's side.

Add file names to cargo metadata

This is very similar to the previous option, and does not solve the profiles problem at all.

Print the resulting file name with --no-run

This is my favorite option :)

• It is naturally forward compatible.

• It's naturally extendable to print more information about the environment which is used to run the process (LD_LIBRARY_PATH, command line flags).

• We almost do this already: with --message-format=json we print the path to binary if it is not fresh. I suggest that we always output this info, even we don't actually rebuild the binary. Perhaps we should use a new, separate message, to make it easier to extend it to support additional and to make it easier for the clients to separate dependencies from the actual end binaries.

• It would be trivial to match cargo commands with binary names: just add --no-run option (I'd like to add this flag to cargo run as well).

• I like that on the client side, you have to actually build the binary before trying to use it. This solves debugging stale binaries problem by construction.

I'll be glad to implement any of these options :)

cc @bruno-medeiros @vojtechkral.

[alexcrichton]

Thanks for summarizing the lay of the land here @matklad! I agree with your analysis and I'd be totally down with extending json messages and such. To take that route it sounds like:

• We should emit messages about paths to binaries even if the target is fresh
• We may be able to avoid cargo run --no-run as that's just the equivalent of cargo build, right?
• Perhaps add messages about LD_LIBRARY_PATH with new entries that need to be added?

[vojtechkral]

Thanks for the summary!

Your proposal sounds good to me... In my usecase (ie. gdb, strace et al.) I could do something like gdb $(cargo build --no-run), which sounds allright. It wouldn't be hard to write a short script or Makefile rule for anything more involved...

[koute]

Yes please!

I recently wrote a Cargo subcommand that needs to know the name of the binaries which are produced after a build, and this was absolutely the biggest pain point I had. Right now I have this really ugly hack where I basically just list all of the files in the target directory that could match whatever the kind of a build I want to run could generate, then I try to build it, and then I see what changed.

I would love to have something like cargo rustc [other args] --print-output-path which would just tell me what the previous invocation with the exactly the same arguments has generated.

with --message-format=json we print the path to binary if it is not fresh. I suggest that we always output this info, even we don't actually rebuild the binary.

In my use case I really don't want to use --message-format=json, simply because I don't want to do the formatting of the messages myself. What I have is a (not so simple) wrapper around cargo rustc, and I just want to pipe anything that cargo rustc produces (error messages, etc.) directly into the user's terminal, but still be able to know what output the invocation has generated.

[malbarbo]

How the proposed approaches works for doc tests (they are ephemeral)? The --with approach would work.

[matklad]

How the proposed approaches works for doc tests (they are ephemeral)? The --with approach would work.

I think neither approach would work out of the box, because it's rustdoc who manages the execution of doctests. That is, any solution we implement in cargo, we need to also implement in rustdoc and then make Cargo to forward command line arguments to rustdoc.

But I think that executing doctests directly is an extremely niche use case, and perhaps we can just not do this?

[matklad]

We may be able to avoid cargo run --no-run as that's just the equivalent of cargo build, right?

I think I want run --no-run for IntelliJ Rust, but this is certainly something we can add or not add later. Let me describe my use case, perhaps it is useful elsewhere.

In IDEA UI the rule in general is "everything you can run, you can debug". So the user may launch command line like cargo run --release --bin foo -- my_program_arg, and then use Shift + F10 to run, and Shift + F9 to debug the program, without configuring debugging separately.

So I want to be able to get the command line for debugging from the command for running.

One way to do this is to add --no-run everywhere, so that Cargo would tell me: "Hey, I've compiled everything, and I would launch this binary with these arguments and in this environment, but I'll let the IDE take over from this point".

Another way would be to parse the command line, find out the equivalent build command and figure the necessary args for the final binary. This parsing of Cargo's command line I would love to avoid, but this is not a major problem.

[alexcrichton]

Oh so the user is typing in cargo run and IntelliJ wants to hijack that? I think it's definitely a good idea to avoid parsing arguments and emulating what Cargo is doing, so for that --no-run seems fine to me.

[vojtechkral]

Side note: Just seeing the literal cargo run --no-run command in the discussion, it really stands out as an obvious oxymoron, something really counter-intuitive and possibly confusing. Perhaps we could move it out of the run command entirely and introduce something like cargo path, cargo runcmd, cargo runenv or somesuch?

[matklad]

Side note: Just seeing the literal cargo run --no-run command in the discussion, it really stands out as an obvious oxymoron, something really counter-intuitive and possibly confusing. Perhaps we could move it out of the run command entirely and introduce something like cargo path or somesuch?

A similar issue is that we have cargo test --no-run in the first place, which does something similar to cargo build actually.

[alexcrichton]

Yeah I wouldn't expect a user to type cargo run --no-run, but having tools be able to insert such a command everywhere seems useful!

[vojtechkral]

Ok, let me try to summarize:

• cargo run --no-run seems to be equivalent to cargo build
• the cargo build --no-run is confusing, build doesn't run stuff anyway. If the options is meant for printing information out, why not call it cargo build --print? It could also optionally take an argument specifying what things you'd like printed, something like ps -o.
• cargo test --no-run is, if I'm getting it right, meant to build target in test configuration but not actually run tests. I'm not sure whether this option belongs in cargo test or if it rather should be something like cargo build --test, but in any case, it seems to me to be orthogonal to this issue.

Correct me if I'm wrong in any of those points...

[malbarbo]

@matklad

But I think that executing doctests directly is an extremely niche use case, and perhaps we can just not do this?

My use case for executing doctests directly is to collect code coverage. Although some says that doctest should not be used in code coverage, some functions have doctests that are good enough and not running them would decrease the coverage.

[alexcrichton]

@vojtechkral did you see @matklad's description above? It sounds like users are typing cargo run and then IntelliJ would have to intepret the command line and translate that to cargo build, whereas slapping on a --no-run would be much eaiser.

And currently cargo build --test isn't equivalent to cargo test --no-run because the former builds one test while the latter builds all tests.

[vojtechkral]

@alexcrichton I must have missed that post or misunderstood it earlier. Sorry.

I'm not convinced that's a good reasoning, though. I don't think it's a good idea to make cargo's interface tailored to IntelliJ. Picture a few months/years down the road a newbie comes along:

• Q: How do I get a Rust binary built and a path to it printed? I need that for XYZ.
• A: Just do cargo run --no-run or cargo build --no-run
• Q: Allright, but wtf?
• A: Yeah, I know, it was designed that way a while ago because it's better for debugging in IntelliJ or somesuch...

Could we instead perhaps separate "no running" and "printing"? Or maybe even better make "printing" imply "not running", where applicable? Ie. --print would make sense with cargo build and it would suppress running with cargo run and cargo test, which would make more sense to me that the other way around...

And currently cargo build --test isn't equivalent to cargo test --no-run because the former builds one test while the latter builds all tests.

cargo build --tests perhaps?

I guess my problem with the --no-run flag is that it basically promotes an interface along the lines of cargo dosomething --wait-dont-actually-do-that-do-something-else.

I hope I'm making sense. Anyway, obvisouly, that's just my $.02, if you guys come to the conclusion --no-run is ok enough, that's fine by me, it's IMO still an improvement over --with.

[matklad]

@vojtechkral totally agree with you that --no-run interface feels ugly if used directly by the user.

I think the core problem is that there are actually two issues we are trying to solve

Issue 1

"I need to find out where the artifacts are"

This issues should be covered by #3319 and #3752. This info is currently available in JSON only, but I think we could expose it in the human readable format if --verbose flag is present. So, the answer to " How do I get a Rust binary built and a path to it printed? I need that for XYZ." would be: "if you need it for one-off thing in human format, run cargo with --verbose flag. If you are integrating cargo with other tools, use --message-format=json".

Issue 2

"What is the command line invocation (that is, binary path, command line arguments, environment variables) that cargo would use for my binary? I would love to intercept it exactly"

This is a rather more specific use case. We need it for IntelliJ Rust, but I imagine other usecases. For example, one can implement a command cargo-dbg with the following interface:

User runs cargo test --test foo bar::baz -- --no-capture and sees a test failure. To debug it, they just type gdb in front: cargo gdb test --test foo bar::baz -- --no-capture. cargo-gdb command then ideally should be to just run the original cargo test --test foo bar::baz -- --no-capture and intercept the test binary invocation.

One way to do it is to add a special argument, --no-run, or --intercept. An interesting alternative would be to set a special environment variable: I think it might be even a better solution, because it won't clutter the command line interface intended for humans, and at the same time would allow tools to avoid even slight modifications of original command line.

[vojtechkral]

One way to do it is to add a special argument, --no-run, or --intercept. An interesting alternative would be to set a special environment variable: I think it might be even a better solution, because it won't clutter the command line interface intended for humans, and at the same time would allow tools to avoid even slight modifications of original command line.

Sounds good to me!

[alexcrichton]

@matklad yeah that sounds pretty reasonable, something like RUSTC for wrapping the compiler and like CARGO_WRAP for wrapping other processes

[matklad]

Ok, I think I'll try that, but only after some time.

First, I need to do a week-long vacation in Paris :)

Then, I'd love to experiment with existing artifact json messages and IntelliJ's debugger: it should be possible to do a lot only with --message-format, and perhaps there are some not yet know problems.

I'll edit the issue description to point to --message-format=json in case someone googles it.

[alexcrichton]

Awesome, thanks @matklad and have fun!

[vojtechkral]

Aside: I wrote a small library that should make it easy* to write cargo wrappers for gdb, strace et al once this issue is solved. Standing by for CARGO_WRAP :)

*) example usage:

extern crate cargo_wrap;

fn main() {
	cargo_wrap::cargo_wrap(|target, mut args| {
		args.push(target);
		(String::from("gdb"), args)
	});
}

[alexcrichton]

cc #3866, a proposed solution

[softprops]

Making a note that exposing a binary name would also be helpful for containerization tools that build the copy binaries into a container

[cbourjau]

I just noticed that I never pinged this issue, but I wrote a cargo plugin a while ago which may be interesting to some people until this issue is resolved upstream: cargo-with

It essentially builds on @matklad initial idea of a cargo-dbg. I don't mean to make shameless advertisement, I just thought it might be relevant to this audience here.

[jonhoo]

How about cargo run --print-path as an alternative to cargo run --no-run?

Having a way to get this information without having to dig through JSON would be great, as I don't know how else to locate a binary in a deployment where users may or may not have set a default binary or CARGO_TARGET_DIR.

[alexkreidler]

Any updates?

[anordal]

For integration test purposes, I couldn't be happier about the new CARGO_BIN_EXE_<name> variable (#7697), released in Cargo 1.43.0.

[ensc]

What is the current recommendation to get the final output name? In my use case, the build consists of two phases:

1. build the rust binary (cargo build)
2. finalize it (objcopy -O binary ...)

Both steps are more or less independent; IDEs or external scripts might have their own idea about the --message-format=json output in the first step so that these information are not available for step 2.

Hence, step 2 needs to determine the path to the binary somehow and atm it works only by guessing and relying on the usual behavior (--> cargo metadata | sed 's!.*"target_directory":"\([^"]\+\)",.*!\1!' + external information about "debug" or "release" + known binary name).

[weihanglo]

What is the current recommendation to get the final output name?

Another workaround with jq. Use at your own risk:

cargo build --message-format=json -q | jq -r 'select(.target.kind[0] == "bin") | .executable'

[ensc]

What is the current recommendation to get the final output name?

Another workaround with jq. Use at your own risk:

cargo build --message-format=json -q | jq -r 'select(.target.kind[0] == "bin") | .executable'

I think, using cargo build to get this information is the wrong way and not applicable. It does the actual build and:

• when doing the build manually, I want human readable output
• when running it under an IDE, the IDE needs the output for other purposes

There should be at least some passive command like cargo metadata which outputs these information. But atm, it

• misses: information about debug vs release directories
• does not accept --target
• does not show the final binary name

[matrach]

It seems that cargo run --no-run can be somewhat emulated with a custom target runner. For example:

cargo --config "target.'cfg(unix)'.runner = 'ls'" run

In contrast to cargo +nightly build --out-dir=out_dir -Z unstable-options, it is a bit racy, as the .cargo-lock only covers the build step. So using cp -t out_dir/ as the runner, while attempting multiple builds in the same target, may copy the wrong binary.