Json CLI Argument

[rob-mur]

Hi

I've been having a go at using CLion for rust with Cucumber, and have ran into an issue.

When tests are ran they append a format=json argument to the inner test harness. This is different to the JSON output required by smartbear for their cucumber for jira integration, an example of what standard lib tests output is below:

{ "type": "suite", "event": "started", "test_count": 0 }
{ "type": "suite", "event": "ok", "passed": 0, "failed": 0, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.0005077 }

This looks relatively simple, and I'm not sure yet which bits are actually required by CLion, but please let me know if anyone has any information. Otherwise over the next little while I'll see if I can get a working fork.

[tyranron]

@rob-mur I cannot really grasp what you're after here. Could you provide a little bit more of context?

Regarding JSON exapmples, aren't they about Cucumber Messages protocol?

[rob-mur]

So I don't think this is related to cucumber but rust more generally. One of the cli options of cargo test is -- --format which clion is using to get JSON output of the tests (just pass fail pieces I think).

The issue is that this cli argument is passed in to the test harness (in this case cucumber) which currently throws an error (at least for me on the latest release).

I'm away from my pc but from memory the error is that "json formatter is only available on the nightly compiler", however changing to nightly doesn't seem to resolve it.

To recreate for yourself, just try running some set of tests with cargo test -- --format=json and see what happens.

[rob-mur]

The exact error I'm getting is

error: The "json" format is only accepted on the nightly compiler

I'll see if I can narrow down where exactly this is happening.

[ilslv]

@rob-mur the problem with CLion is, that it's always expects default test harness which has --format=json option. Our crate requires you to specify harness = false to be able to hijack cargo test command. This article may also help.
For example you can specify CLion cargo configuration to run test --test wait -- --tags='@any', but resulting command will be cargo test --color=always --test wait --no-fail-fast -- --format=json --tags='@any' -Z unstable-options --show-output. The solution would be to use Cucumber::with_cli but instead of before_time, use format to skip this argument.

[rob-mur]

Thanks for the response. I'll have a go later and see what the result is with that argument skipped as you suggest.

It may be however that this output is required to integrate properly with the IDE and then we could look in to taking this argument and producing an equivalent output?

[ilslv]

@rob-mur

It may be however that this output is required to integrate properly with the IDE and then we could look in to taking this argument and producing an equivalent output?

I'll look into that and maybe implement custom Writer for that purpose if it's possible.

[rob-mur]

@rob-mur

It may be however that this output is required to integrate properly with the IDE and then we could look in to taking this argument and producing an equivalent output?

I'll look into that and maybe implement custom Writer for that purpose if it's possible.

I've had a go, and using a custom cli struct for format, Z and show-output, you can get CLion to run the tests.

This is already a great step forward as it allows you to utilise the CLion debugger in tests so many thanks for that!

My gut feeling was right though in that as far as the IDE is concerned there are no test results so it doesn't quite integrate as expected.

Feel free to have a go if you like but it's not particularly high priority!

[tyranron]

@ilslv given #203 and #218 I think this now diserves to be implemented on our side behind a "clion" feature gate.

[tyranron]

@rob-mur released in 0.14.0.