Input on a json scema for bench reporting

[epage]

In a theoretical future where test/bench binaries are encouraged to report back results to cargo as json which then formats and reports them, what would bencher see as needed for the json schema?

I've been considering how we can improve testing in Rust and one area I've found is the limited contract between cargo and test harnesses is a source of friction. I'm also hoping that this lens for looking at json reporting can help get the schema stabilized.

For reference, this is the current schema for benches.

[epompeii]

First off, thank you for taking this on @epage (even if it is at the bottom of your list)!
Having a stable JSON schema for cargo test and cargo bench would be great.

I'm definitely not an expert here, but I find the current libtest bench schema to be very... confusing.
libtest bench only collects the median, but it then it reports the deviation.
I assume that deviation refers to the standard deviation. And if that is the case, we should instead be reporting inter-quartile range as the bounds for median.
Better yet, we could add both mean (to pair with the existing deviation) and iqr (to pair with the existing median) as well as both min and max values.
Yet another point of "holy envy" for the pytest ecosystem: https://github.com/ionelmc/pytest-benchmark#screenshots
It may be good to loop in @bheisler here.

From a purely Bencher-centric perspective, having cargo bench output only JSON when in JSON mode would be ideal. That is, it wouldn't include any of the plain text around the JSON. Also, having all output contained within a single object or array would ease parsing. Some of the current unstable outputs are a newline delimited list of JSON objects, which is less than ideal.

[epompeii]

I similarly reached out to criterion: bheisler/criterion.rs#693

Ah, great! Are you considering supporting non-wall clock time based benchmarks directly or by extension points in the schema?

Thanks for the pointer on that; we should look at its json output. This is also reminding me that we should also look at hyperfine's json output.

This is the pytest-benchmark JSON output argument: --benchmark-json results.json
They store their JSON results to a file upon completion.

--benchmark-json=PATH
Dump a JSON report into PATH. Note that this will include the complete data (all the timings, not just the stats).

This is the hyperfine JSON output schema:

• https://github.com/sharkdp/hyperfine/blob/master/src/export/json.rs
• https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/benchmark_result.rs#L11

Both pytest-benchmark and hyperfine along with many other benchmark harnesses I've instrumented use the JSON output as a way to sort of dump/export a lot more data than what is usually presented. This often includes all measurements that were collected.

Part of this will involve a cargo test --format json flag so all output will be in json, rather than mixed.

Great!

However, ndjson will be continued to be used. ...

That totally makes sense. I'm currently not doing any streaming in Bencher. Though it definitely makes sense to preserve the streaming use case.
This seems to point to two separate kinds of "JSON output" though:

1. Streaming output in ndjson format, containing statistics only
2. A dump in JSON format of all measurements and statistics

For Bencher, either would work. Though, I can see both being of use to others under different conditions.

[epage]

Ah, great! Are you considering supporting non-wall clock time based benchmarks directly or by extension points in the schema?

I think they should be supported directly. I see extensions being used for providing additional data.

Both pytest-benchmark and hyperfine along with many other benchmark harnesses I've instrumented use the JSON output as a way to sort of dump/export a lot more data than what is usually presented. This often includes all measurements that were collected.

Thanks for digging up that info!

I've been leaning towards that approach as well. My main concern with it is if that would be too many samples in some cases.

For Bencher, either would work. Though, I can see both being of use to others under different conditions.

We would likely be aiming for our API to be minimal, so we'd likely only support one mode, especially if people can derive the other from it.

[epompeii]

I think they should be supported directly. I see extensions being used for providing additional data.

Okay, if that is the case it may be worth checking out the list of profilers that rustc-perf uses: https://github.com/rust-lang/rustc-perf/tree/master/collector#profiling
I haven't done a comprehensive survey of them yet, but I imagine there is a reasonable superset of the measures collected.
Though one thing that I will note in working with iai is that I think the calculation of "Estimated Cycles" should be configurable (https://github.com/bheisler/iai/blob/main/src/lib.rs#L264), but maybe that's more of a harness specific concern?

I've been leaning towards that approach as well. My main concern with it is if that would be too many samples in some cases.

This is spot on. I actually had to increase the default max request size in Bencher because of the size of the pytest-benchmark JSON output.

We would likely be aiming for our API to be minimal, so we'd likely only support one mode, especially if people can derive the other from it.

That totally makes sense. I guess the point of tension is that the underlying samples are not derivable from the summary statistics. Currently, I'm just targeting the lowest common denominator across benchmark harnesses, so I don't collect individual samples, just summary statistics. However, it's not inconceivable that in the future I may want to be able to provide a regression plot within Bencher similar to Criterion: https://bheisler.github.io/criterion.rs/book/user_guide/plots_and_graphs.html#regression

Maybe there is just a single, streaming JSON mode (ndjson) with an optional samples key that could be toggled with an --all-samples flag. When enabled, it would include all samples used to generate the summary statistic(s).

[epage]

Maybe there is just a single, streaming JSON mode (ndjson) with an optional samples key that could be toggled with an --all-samples flag. When enabled, it would include all samples used to generate the summary statistic(s).

On the other hand, I'm concerned about trying to lock-in what the minimally required statistical analysis is...

[epompeii]

From what I've seen, the Pareto principle/vital few are:

• mean with standard deviation
• median with interquartile range
• minimum and maximum values

Most benchmark harnesses output one or both of the first two: https://bencher.dev/docs/explanation/adapters
And some also include the min/max.

Though Criterion uses median absolute deviation instead of interquartile range: https://bheisler.github.io/criterion.rs/book/user_guide/command_line_output.html#additional-statistics
Criterion also calculates slope and uses "slope (if available) or the mean (if not)": https://bheisler.github.io/criterion.rs/book/cargo_criterion/external_tools.html#json-messages