benchmark-checklist.tex

\chapter{Benchmark Checklist}
\label{sec:benchmark-checklist}

We expect LDBC benchmarks to be used in many scenarios.
For most research papers, fully audited results are unrealistic and even unaudited results can provide insight into the performance of the systems under test (SUT). However, we ask authors to include the following information in their papers:

\begin{itemize}
\item Were the results cross-validated for at least one scale factor?
\item Were the results cross-validated for all scale factors used in the benchmark?
\item Does the SUT have a persistent storage?
\item Does the SUT provide ACID transactions?
\item Does the SUT provide any level of fault-tolerance?
\item How many warm-up rounds were performed?
\item How many execution rounds were performed?
\item How were the execution times summarized?\footnote{Paper~\cite{DBLP:conf/sc/HoeflerB15} provides an excellent overview on how to summarize benchmark results.}
\item Is the loading phase included in the query execution times?\footnote{This might be relevant for systems without persistent storage, or systems providing lazy/incremental computation.}
\item If the SUT is not your own system, did you contact its developers or experts to help optimizing the queries?\footnote{For a research prototype tool, the tuning knobs are usually not well documented. Hence, it is worth contacting the tool's authors, who are generally keen to help. For more mature systems (\eg most established RDBMSs), there is a large body of knowledge available, in the form of books and online forums, which should help your optimization efforts. It is also possible to contact experienced DBAs who can assist with fine tuning the system.}
\end{itemize}

These results will help the reader to put the results in context. For example, a non-ACID compliant, non-fault-tolerant system working on read-only graphs and offering no persistent storage is expected to have significantly better results than a fully-fledged disk-based DBMS.

We also suggest the reader to take a look at the checklist presented in~\cite{DBLP:conf/sigmod/RaasveldtHGM18}.