JMH with OpenJ9

[apete]

I get these warnings:

WARNING: Not a HotSpot compiler command compatible VM ("Eclipse OpenJ9 VM-11.0.2"), compilerHints are disabled.

WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.

Running the benchmark seems to work, but can I trust the numbers?

I'm running benchmarks to compare HotSpot and OpenJ9.

[dsouzai]

I asked our test team, and they said that the JMH Framework can be used. However, the microbenchmark tests that come with the framework may be specific to Hotspot; you'd have to take a look at each one individually to see what they measure.

Another alternative framework you could use to compare HotSpot and OpenJ9 is BumbleBench https://github.com/AdoptOpenJDK/bumblebench .

[apete]

Thanks!

In that case maybe you're interested in the results: https://ojalgo.blogspot.com/2019/02/quick-test-to-compare-hotspot-and-openj9.html

I have some existing JMH benchmarks that I've used to compare pure java linear algebra libraries. Now I'm interested in using those to compare JVM:s instead.

[dsouzai]

Those results are interesting, thanks @apete ; I'll try to give the benchmark a spin at some point and see if there's anything that stands out in terms of why there's so much variability on the OpenJ9 runs.

[apete]

The variability of OpenJ9 is consistent... I've executed other benchmarks as well. Comparing the results with what I'm used to see, performance is fluctuating.

I updated that blog post with results for the latest available community edition of Graal included as well. (Unfortunately it was based on OpenJDK 8.)

I also added links to the raw benchmark results if anyone want to verify what's shown in the charts.

I'll close this issue now as my original question has been answered.

[andrewcraik]

Just an additional comment on JMH - if your benchmarks use the CompilerOptions to change how HotSpot compiles then those configurations won't apply to OpenJ9 which could lead to different results. The ACM version of the bench is clearly not working as well as it might given the perf. I am looking at it and will create an issue here on the OpenJ9 project where we can make notes about how to make it better.

[apete]

FYI

I did a follow-up comparison with HotSpot, Graal CE and Graal EE. While HotSpot is a natural benchmark now. Going forward Graal EE is the one to beat.

https://ojalgo.blogspot.com/2019/02/oracles-jvms-hotspot-graal-ce-graal-ee.html