Feature request: interleaved execution

[bhansconnect]

Currently hyperfine will benchmark each command in order. For example, hyperfine -m 100 ./command1 ./command2 will run command1 100 times and then run command2 100 times. I believe that it would be very useful to be able to request interleaved execution, where instead hyperfine would run command1 then comand2 then command1 then command2 until both have been run 100 times.

I have used hyperfine for measuring the performance of a number of quite sensitive processes where background noise or even execution order can decide which process will be faster. If run first, the cpu is not that hot and this will run faster, but if run second, the cpu is hot and it will run slower. By interleaving the executions, both processes end up evenly getting effected by these forms of background noise. This enables the use of hyperfine in these situations.

I understand that it would be better to run on a machine with less noise, or to warmup for long enough that both are running at full heat, but in many cases, that would take significantly more time and effort than just running the processes interleaved.

Hopefully this is a feature that overall makes sense and seems valuable. Depending on how hyperfine is implemented, it may be rather trivial to add this feature (this is what I am hoping for). I am willing to help and contribute this feature if it is acceptable to add to hyperfine, but could probably use some pointers given I have never looked at the code base.

As an extra note, with interleaved execution, it would theoretically be best if the order of the commands is changed every iteration to try and control for the effect that ordering may cause.

[sharkdp]

Thank you very much for your detailed feature request. I believe this is a duplicate of #21? Maybe we could continue the discussion there.

I'd be very interested in a reproducible test setup. I never experienced any problems with benchmarks myself that couldn't easily be solved by something like --warmup 3.

I understand that it would be better to run on a machine with less noise, or to warmup for long enough that both are running at full heat, but in many cases, that would take significantly more time and effort than just running the processes interleaved.

"Significantly more effort" being the time to add --warmup 3, or what do you have in mind?

"Significantly more time"... because each command invocation takes a long time and the warmup runs would take too long?

As an extra note, with interleaved execution, it would theoretically be best if the order of the commands is changed every iteration to try and control for the effect that ordering may cause.

Good point. That could be interesting to evaluate. Another problem with interleaved execution is the placement of "setup" and "cleanup" commands. Currently, they are being executed once before each benchmark run:

For interleaved execution, we would have to run a "setup, prepare, cmd, cleanup" series for each execution, right? Doing two consecutive runs for each command would optimize this a bit to:

setup1, prepare1, cmd1, prepare1, cmd1, cleanup1
setup2, prepare2, cmd2, prepare2, cmd2, cleanup2
…

Hopefully this is a feature that overall makes sense and seems valuable. Depending on how hyperfine is implemented, it may be rather trivial to add this feature (this is what I am hoping for).

It's probably not super hard, but also not trivial. Everything is currently structured around the idea that benchmarks are independent (except for the benchmark comparison that is being printed at the end). Interleaved execution would also require a redesign of the progress-bar output that is currently being shown independently for each benchmark.

I am willing to help and contribute this feature if it is acceptable to add to hyperfine, but could probably use some pointers given I have never looked at the code base.

Sounds great! I would like to discuss all the implications first though. Also, I'm in the middle of a large-scale refactoring of the source code (#482 is just one part of many), so this might not be the best time to start a new feature 😄.

On the other hand, it should be significantly easier to make changes to the code base after this work has been finished. I also added a huge number of integration tests.

Since this interleaving option seems to be a popular request, I will try to keep it in mind when further restructuring the code.

[bhansconnect]

Wow. Sorry I missed the duplication. I scanned the issues quickly but didn't see it. I'll add more comments on the other issue.