benchdb - A database and query tool for JMH results
When you run benchmarks with JMH you usually look at the results table printed after the run or maybe generate a JSON file and feed it into JMH Visualizer for immediate consumption. This approach does not scale well when you benchmark lots of different changes or want to compare historical data or visualize more complex benchmark results graphically.
benchdb takes a JMH result file plus some captured environment data (platform, Java environment, git data for git-based projects) and stores it in a relational database of your choice. You can later list and retrieve these results and run queries over a single result or combining multiple results. Thanks to the stored environment data you always know when, what and where you ran a benchmark.
-
Clone benchdb and run
sbt core/publishLocal
. -
Create the command line app with Coursier, e.g.:
> cs bootstrap com.lightbend.benchdb::benchdb-core:latest.release com.h2database:h2:1.4.200 -o benchdb
Note that the second dependency is the database JDBC driver, in this case an H2 embedded database which is recommended for local use on a single machine. You should replace this with the database driver of your choice (e.g.
mysql:mysql-connector-java:8.0.19
for MySQL/MariaDB). benchdb doesn't use any advanced database features. Any database supported by Slick should work. -
In order to use an embedded H2 database you can run
benchdb create-config
to create a default configuration file.benchdb.conf
in your home directory. benchdb config files use HOCON syntax for Typesafe Config. You can also specify a path for additional configurations or ignore the user config file with command line options.For a multi-machine setup with a database server, you need to create the configuration manually. It needs to contain at least a DatabaseConfig for Slick under the name
db
. Here is an example for connecting to a MySQL/MariaDB server:db { profile = "slick.jdbc.MySQLProfile$" db { connectionPool = disabled dataSourceClass = "com.mysql.cj.jdbc.MysqlDataSource" properties = { serverName = "hostname" portNumber = "3307" databaseName = "benchdb" user = "benchdb" password = "password" serverTimezone = "UTC" } } }
-
Run
benchdb init-db --force
to initialize the database schema.
-
benchdb --help
shows the list of supported commands. Specifying a command name followed by--help
shows further options and parameters for that command. -
First you need to insert some benchmark results into the database, e.g.:
> benchdb insert-run --project-dir ../scala --msg "Test 1" --jmh-args "-wm bulk" ../scala/test/benchmarks/jmh-result.json
The specified project directory (default: current directory) is used to determine git environment data.
-
benchdb list
lists the benchmark runs in the database, e.g.:> benchdb list --git-data /----+---------------------+----------+---------+---------------------+----------------------------------+------------------------------------\ | ID | Timestamp | Msg | Git SHA | Git Timestamp | Git Origin | Git Upstream | |----+---------------------+----------+---------+---------------------+----------------------------------+------------------------------------| | 4 | 2020-03-24 14:15:15 | Test 3 | d335189 | 2020-03-09 16:06:23 | git@github.com:szeiger/scala.git | https://github.com/scala/scala.git | | 3 | 2020-03-24 14:14:57 | Test 3 | d335189 | 2020-03-09 16:06:23 | git@github.com:szeiger/scala.git | https://github.com/scala/scala.git | | 2 | 2020-03-24 13:18:38 | Test 2 | d335189 | 2020-03-09 16:06:23 | git@github.com:szeiger/scala.git | https://github.com/scala/scala.git | | 1 | 2020-03-24 13:07:30 | Test job | d335189 | 2020-03-09 16:06:23 | git@github.com:szeiger/scala.git | https://github.com/scala/scala.git | \----+---------------------+----------+---------+---------------------+----------------------------------+------------------------------------/ 4 test runs found.
-
benchdb results
generates a table similar to the one produced by JMH itself. You can specify one or more run IDs to show (defaulting to the latest run if no ID is given) and also filter benchmark names with glob patterns:> benchdb results -r1 -b*100p* /----------------------------------------------------------+--------+------+-----+------------+----------+-------\ | Benchmark | (size) | Mode | Cnt | Score | Error | Units | |----------------------------------------------------------+--------+------+-----+------------+----------+-------| | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 1 | avgt | 20 | 9.046 | 0.073 | ns/op | | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 10 | avgt | 20 | 11.558 | 0.099 | ns/op | | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 100 | avgt | 20 | 503.222 | 11.211 | ns/op | | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 1000 | avgt | 20 | 6163.309 | 278.645 | ns/op | | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 10000 | avgt | 20 | 41181.090 | 1407.833 | ns/op | | scala.collection.immutable.VectorBenchmark2.nvFilter100p | 50000 | avgt | 20 | 195477.388 | 4077.424 | ns/op | \----------------------------------------------------------+--------+------+-----+------------+----------+-------/
Secondary metrics can be used in place of the primary metric with -metric
, (for instance, -metric ·gc.alloc.rate.norm
shows the the per-operation allocations recorded by JMH's -prof gc
.
-
Extractor patterns can be used to extract additional parameters from benchmark names. They are glob patterns with regular expression-like capture groups. Unnamed groups are discarded, named groups are extracted into parameters. For example:
> benchdb results -r1 --extract (*2.)nvFilter(percent=*)p /--------------------+--------+-----------+------+-----+------------+----------+-------\ | Benchmark | (size) | (percent) | Mode | Cnt | Score | Error | Units | |--------------------+--------+-----------+------+-----+------------+----------+-------| | nvFilter(percent)p | 1 | 0 | avgt | 20 | 8.824 | 0.381 | ns/op | | nvFilter(percent)p | 10 | 0 | avgt | 20 | 8.902 | 0.075 | ns/op | | nvFilter(percent)p | 100 | 0 | avgt | 20 | 40.800 | 0.544 | ns/op | | nvFilter(percent)p | 1000 | 0 | avgt | 20 | 134.629 | 1.996 | ns/op | | nvFilter(percent)p | 10000 | 0 | avgt | 20 | 1712.683 | 31.730 | ns/op | | nvFilter(percent)p | 50000 | 0 | avgt | 20 | 8186.502 | 130.088 | ns/op | | nvFilter(percent)p | 1 | 100 | avgt | 20 | 9.046 | 0.073 | ns/op | | nvFilter(percent)p | 10 | 100 | avgt | 20 | 11.558 | 0.099 | ns/op | | nvFilter(percent)p | 100 | 100 | avgt | 20 | 503.222 | 11.211 | ns/op | | nvFilter(percent)p | 1000 | 100 | avgt | 20 | 6163.309 | 278.645 | ns/op | | nvFilter(percent)p | 10000 | 100 | avgt | 20 | 41181.090 | 1407.833 | ns/op | | nvFilter(percent)p | 50000 | 100 | avgt | 20 | 195477.388 | 4077.424 | ns/op | | nvFilter(percent)p | 1 | 50 | avgt | 20 | 13.321 | 0.302 | ns/op | | nvFilter(percent)p | 10 | 50 | avgt | 20 | 62.017 | 1.439 | ns/op | | nvFilter(percent)p | 100 | 50 | avgt | 20 | 595.161 | 34.352 | ns/op | | nvFilter(percent)p | 1000 | 50 | avgt | 20 | 5951.751 | 56.474 | ns/op | | nvFilter(percent)p | 10000 | 50 | avgt | 20 | 43305.200 | 1221.533 | ns/op | | nvFilter(percent)p | 50000 | 50 | avgt | 20 | 196567.912 | 8389.588 | ns/op | \--------------------+--------+-----------+------+-----+------------+----------+-------/
-
You can then pivot one or more parameters to compare their results side by side:
> benchdb results -r1 --extract (*2.)nvFilter(percent=*)p --pivot percent /--------------------+--------+------+-----+----------+---------+------------+----------+------------+----------+-------\ | (percent) | | | | 0 | 50 | 100 | | | Benchmark | (size) | Mode | Cnt | Score | Error | Score | Error | Score | Error | Units | |--------------------+--------+------+-----+----------+---------+------------+----------+------------+----------+-------| | nvFilter(percent)p | 1 | avgt | 20 | 8.824 | 0.381 | 13.321 | 0.302 | 9.046 | 0.073 | ns/op | | nvFilter(percent)p | 10 | avgt | 20 | 8.902 | 0.075 | 62.017 | 1.439 | 11.558 | 0.099 | ns/op | | nvFilter(percent)p | 100 | avgt | 20 | 40.800 | 0.544 | 595.161 | 34.352 | 503.222 | 11.211 | ns/op | | nvFilter(percent)p | 1000 | avgt | 20 | 134.629 | 1.996 | 5951.751 | 56.474 | 6163.309 | 278.645 | ns/op | | nvFilter(percent)p | 10000 | avgt | 20 | 1712.683 | 31.730 | 43305.200 | 1221.533 | 41181.090 | 1407.833 | ns/op | | nvFilter(percent)p | 50000 | avgt | 20 | 8186.502 | 130.088 | 196567.912 | 8389.588 | 195477.388 | 4077.424 | ns/op | \--------------------+--------+------+-----+----------+---------+------------+----------+------------+----------+-------/
-
benchdb chart
generates line charts (using the Google Charts library). The parameters are the same as forresults
. Charts require a single free parameter which must beLong
-valued (i.e. all instances can be parsed into aLong
-- the actual types of the benchmark parameters are not preserved by JMH; it stores everything as a string), likesize
in the example above. In case of pivoted results, all pivoted columns are rendered together as individual series in a single chart. The result ofbenchdb chart
is a single, self-contained HTML file. If no output file is specified, it is written to a temporary file and opened in the default browser.
benchdb is NOT supported under the Lightbend subscription.
Contributions to this project are very welcome!