-
Notifications
You must be signed in to change notification settings - Fork 125
Overview
A project is a software project that has a version control repository. Within that repository will be multiple branches, each of which contains a sequence of revisions.
An executable is a compilation of a project, which benchmarks are run against. The concept of executables historically comes from testing multiple implementations of Python - CPython, PyPy, etc. In that context, there were multiple executables that had been compiled from different projects (cpython/pypy), with different versions (cpython 2.6.2, pypy 1.3, 1.4, 1.5, latest) and different compilation options (cpython can optionally use psyco-profile, pypy can optionally use a jit). Each of these executables were meant to be interchangeable...
Certain revisions can optionally be given a tag. They then make the results from the executables of their project available as baselines to compare other results against on the timeline and comparison screens.
A benchmark is a particular test that returns a metric on performance. The benchmark specifies the units of that metric, and whether less is better (e.g. for time, a lower number of seconds is usually optimal), or not (e.g. for throughput). Benchmarks come in two kinds: cross-project and own-project. Only cross-project benchmarks are shown on the comparison screen.
An environment is a particular context in which tests are executed - a machine with a given CPU, operating system, and speed. Recording this is significant to ensure that comparison between results is meaningful.
The results of running a particular benchmark in a particular environment, on an executable compiled from a particular revision of a project, are uploaded to the server. Each result must have a value, and can optionally also have a minimum, maximum, and standard deviation.
Results are grouped into reports that cover all the benchmarks on a particular revision and executable.
This screen can be used to monitor the detailed impact a set of revisions has on the benchmark results, as well as browsing those results.
Only projects that are configured to track changes can be selected. Any executable from those projects can be chosen, and a list of recent revisions with benchmark results is presented for selection.
The benchmark results are presented along with the change from the previous set of results, revision info and the commit logs since the previous revision
This shows how benchmark results have varied over time. Only results from the default project are shown, but they can be normalized against any of the valid tagged baseline revisions from any project.
This compares results across executables (and therefore projects). Possible executables are the latest version on each branch of each project (including the default), plus any tagged baseline revisions.
Benchmarks are grouped by units. Only benchmarks marked as cross-project are shown.
Users can select a subset of executables, benchmarks and environments from those available. There are also some useful statistical options to normalize against a particular executable (including the latest) as baseline, and display the charts as stacked bars or relative bars.