EvoMaster Benchmark (EMB): a set of web/enterprise applications for scientific research in Software Engineering.
We collected several different systems running on the JVM, in different programming languages such as Java and Kotlin. In this documentation, we will refer to these projects as System Under Test (SUT). Currently, the SUTs are either REST, GraphQL or RPC APIs.
For each SUT, we implemented driver classes, which can programmatically start, stop and reset the state of SUT (e.g., data in SQL databases). As well as enable setting up different properties in a uniform way, like choosing TCP port numbers for the HTTP servers. If a SUT uses any external services (e.g., a SQL database), these will be automatically started via Docker in these driver classes.
This collection of SUTs was originally assembled for easing experimentation with the fuzzer called EvoMaster. However, finding this type of application is not trivial among open-source projects. Furthermore, it is not simple to sort out all the technical details on how to set these applications up and start them in a simple, uniform approach. Therefore, this repository provides the important contribution of providing all these necessary scripts for researchers that need this kind of case study.
NOTE: version 1.6.1 was last one in which we still updated drivers for JavaScript and C#. Those SUTs are not built anymore by default, and latest versions of EvoMaster might not work on those old drivers. Updating drivers for different programming languages (and re-implement white-box heuristics) is a massive amount of work, which unfortunately has little to no value for the scientific community (based on our experience). Those SUTs are still here in EMB to enable black-box experiments (and to be able to replicate old experiments), but unfortunately not for white-box testing with latest versions of EvoMaster.
A video providing some high level overview of EMB can be found here.
All the code that is new for this repository (e.g., the driver classes) is released under Apache 2.0 license. However, this repository contains as well sources from different open-source projects, each one with its own license, as clarified in more details beneath.
To see an example of using these drivers with EvoMaster to generate test cases, you can look at this short video (5 minutes).
If you are using EMB in an academic work, you can cite the following:
A. Arcuri, M. Zhang, A. Golmohammadi, A. Belhadi, J. P. Galeotti, B. Marculescu, S. Seran.
EMB: A Curated Corpus of Web/Enterprise Applications And Library Support for Software Testing Research.
In IEEE International Conference on Software Testing, Validation and Verification (ICST), 2023.
The projects were selected based on searches using keywords on GitHub APIs, using convenience sampling. Several SUTs were looked at, in which we discarded the ones that would not compile, would crash at startup, would use obscure/unpopular libraries with no documentation to get them started, are too trivial, student projects, etc. Where possible, we tried to prioritize/sort based on number of stars on GitHub.
Note that some of these open-source projects might be no longer supported, whereas others are still developed and updated. Once a system is added to EMB, we do not modify nor keep it updated with its current version under development. The reason is that we want to keep an easy to use, constant set of case studies for experimentation that can be reliably used throughout the years.
The SUTs called NCS (Numerical Case Study) and SCS (String Case study) are artificial, developed by us. They are based on numerical and string-based functions previously used in the literature of unit test generation. We just re-implemented in different languages, and put them behind a web service.
For the RESTful APIs, each API has an endpoint where the OpenAPI/Swagger schemas can be downloaded from. For simplicity, all schemas are also available as JSON/YML files under the folder openapi-swagger.
More details (e.g., #LOCs and used databases) on these APIs can be found in this table.
-
Familie Ba Sak (MIT), jdk_17_maven/cs/rest/familie-ba-sak, from https://github.com/navikt/familie-ba-sak
-
Payments Public API (MIT), jdk_11_maven/cs/rest/pay-publicapi, from https://github.com/alphagov/pay-publicapi
-
Session Service (not-known license), jdk_8_maven/cs/rest/original/session-service, from https://github.com/cBioPortal/session-service
-
Bibliothek (MIT), jdk_17_gradle/cs/rest/bibliothek, from https://github.com/PaperMC/bibliothek
-
Reservations API (not-known license), jdk_11_gradle/cs/rest/reservations-api, from https://github.com/cyrilgavala/reservations-api
-
Genome Nexus (MIT), jdk_8_maven/cs/rest-gui/genome-nexus, from https://github.com/genome-nexus/genome-nexus
-
Market (MIT), jdk_11_maven/cs/rest-gui/market, from https://github.com/aleksey-lukyanets/market
-
Features-Service (Apache), jdk_8_maven/cs/rest/original/features-service, from https://github.com/JavierMF/features-service
-
Scout-API (MIT), jdk_8_maven/cs/rest/original/scout-api, from https://github.com/mikaelsvensson/scout-api
-
ProxyPrint (Apache), jdk_8_maven/cs/rest/original/proxyprint, from https://github.com/ProxyPrint/proxyprint-kitchen
-
CatWatch (Apache), jdk_8_maven/cs/rest/original/catwatch, from https://github.com/zalando-incubator/catwatch
-
OCVN (MIT), jdk_8_maven/cs/rest-gui/ocvn, from https://github.com/devgateway/ocvn
-
News (LGPL), jdk_8_maven/cs/rest/artificial/news, from https://github.com/arcuri82/testing_security_development_enterprise_systems
-
Restcountries (MPL), jdk_8_maven/cs/rest/original/restcountries, from https://github.com/apilayer/restcountries
-
Languagetool (LGPL), jdk_8_maven/cs/rest/original/languagetool, from https://github.com/languagetool-org/languagetool
-
CWA-Verification-Server (Apache), jdk_11_maven/cs/rest/cwa-verification-server, from https://github.com/corona-warn-app/cwa-verification-server
-
Gestao Hospital (not-known license), jdk_8_maven/cs/rest-gui/gestaohospital, from https://github.com/ValchanOficial/GestaoHospital
-
NCS, jdk_8_maven/cs/rest/artificial/ncs, (not-known license, artificial numerical examples coming from different sources)
-
SCS, jdk_8_maven/cs/rest/artificial/scs, (not-known license, artificial string examples coming from different sources)
-
Spring-Pet-Clinic (Apache), jdk_8_maven/cs/graphql/petclinic-graphql, from https://github.com/spring-petclinic/spring-petclinic-graphql
-
Patio-Api (GPL), jdk_11_gradle/cs/graphql/patio-api, from https://github.com/patio-team/patio-api
-
Timbuctoo (GPL), jdk_11_maven/cs/graphql/timbuctoo, from https://github.com/HuygensING/timbuctoo
-
NCS, jdk_8_maven/cs/graphql/graphql-ncs, (not-known license, artificial numerical examples coming from different sources)
-
SCS, jdk_8_maven/cs/graphql/graphql-scs, (not-known license, artificial string examples coming from different sources)
-
Signal-Registration (not-known license), jdk_17_maven/cs/grpc/signal-registration, from https://github.com/signalapp/registration-service
-
NCS (not-known license, artificial numerical examples coming from different sources). Thrift: jdk_8_maven/cs/rpc/thrift/artificial/thrift-ncs. gRPC: jdk_8_maven/cs/rpc/grpc/artificial/grpc-ncs.
-
SCS (not-known license, artificial string examples coming from different sources). Thrift: jdk_8_maven/cs/rpc/thrift/artificial/thrift-scs. gRPC: jdk_8_maven/cs/rpc/grpc/artificial/grpc-scs.
- Spring-PetClinic (Apache), jdk_17_maven/cs/web/spring-petclinic, from https://github.com/spring-projects/spring-petclinic
-
Disease-sh-API (GPL), js_npm/rest/disease-sh-api, from https://github.com/disease-sh/API
-
Cyclotron (MIT), js_npm/rest/cyclotron, from https://github.com/ExpediaInceCommercePlatform/cyclotron
-
SpaceX-API (Apache-2.0 License), js_npm/rest/spacex-api, from https://github.com/r-spacex/SpaceX-API
-
Realworld-App (ISC), js_npm/rest/realworld-app, from https://github.com/lujakob/nestjs-realworld-example-app
-
NCS, js_npm/rest/ncs, (not-known license, artificial numerical examples coming from different sources)
-
SCS, js_npm/rest/scs, (not-known license, artificial string examples coming from different sources)
-
Menu.API (not-known license), from https://github.com/chayxana/Restaurant-App
-
SampleProject (MIT), from https://github.com/kgrzybek/sample-dotnet-core-cqrs-api
-
NCS (not-known license, artificial numerical examples coming from different sources)
-
SCS (not-known license, artificial string examples coming from different sources)
-
React-Finland (not-known license), js_npm/graphql/react-finland, from https://github.com/ReactFinland/graphql-api
-
E-Commerce Server (MIT), js_npm/graphql/ecommerce-server, from https://github.com/react-shop/react-ecommerce
Due to several reasons, the software in this repository is not published as a library (e.g., on Maven and NPM). To use EMB, you need to clone this repository:
git clone https://github.com/EMResearch/EMB.git
There are 2 main use cases for EMB:
-
Run experiments with EvoMaster
-
Run experiments with other tools
Everything can be setup by running the script scripts/dist.py
.
Note that you will need installed at least Maven, Gradle, JDK 8, JDK 11, JDK 17, NPM, as well as Docker.
Also, you will need to setup environment variables like JAVA_HOME_8
, JAVA_HOME_11
and JAVA_HOME_17
.
The script will issue error messages if any prerequisite is missing.
Once the script is completed, all the SUTs will be available under the dist
folder, and a dist.zip
will be created as well (if scripts/dist.py
is run with True
as input).
Regarding Maven, most-third party dependencies are automatically downloaded from Maven Central.
However, some dependencies are from GitHub, which unfortunately require authentication to be able to download such dependencies.
Unfortunately, they have no intention to fix this huge usability issue :(
In your home folder, you need to create a configuration file for Maven, in particular .m2/settings.xml
, with the following configurations:
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>github</id>
<!-- Old pre Maven 3.9.0 version -->
<username>YOURUSERNAME</username>
<password>???</password>
<!-- New post Maven 3.9.0 version -->
<configuration>
<httpHeaders>
<property>
<name>Authorization</name>
<value>Bearer ???</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
Which configuration to use depends on the version of Maven (it was changed in version 3.9.0).
In latest versions of Maven, you need to create an authorization token in GitHub (see more info directly on GitHub documentation pages), and put it instead of ???
.
In the built dist
folder, the files will be organized as follows:
<name>-sut.jar
will be the non-instrumented SUTs, whereas their executable drivers will be called <name>-evomaster-runner.jar
.
Instrumentation can be done at runtime by attaching the evomaster-agent.jar
JavaAgent. If you are running experiments with EvoMaster, this will be automatically attached when running experiments with exp.py
(available in the EvoMaster's repository). Or it can be attached manually with JVM option -Devomaster.instrumentation.jar.path=evomaster-agent.jar
when starting the driver.
For running experiments with EvoMaster, you can also "start" each driver directly from an IDE (e.g., IntelliJ). Each of these drivers has a "main" method that is running a REST API (binding on default port 40100), where each operation (like start/stop/reset the SUT) can be called via an HTTP message by EvoMaster.
You can also build (and install) each module separately, based on needs. For example, a Maven module can be installed with:
mvn clean install -DskipTests
However, it is important to understand how this repository is structured, to be able to effectively navigate through it.
Each folder represents a set of SUTs (and drivers) that can be built using the same tools.
For example, the folder jdk_8_maven
contains all the SUTs that need JDK 8 and are built with Maven.
On the other hand, the SUTs in the folder jdk_11_gradle
require JDK 11 and Gradle.
For thr JVM, each module has 2 submodules, called cs
(short for "Case Study") and em
(short for "EvoMaster").
cs
contains all the source code of the different SUTs, whereas em
contains all the drivers.
Note: building a top-module will build as well all of its internal submodules.
The driver classes for Java are called EmbeddedEvoMasterController
.
Note that Java also has a different kind of driver called ExternalEvoMasterController
.
The difference is that in External the SUT is started on a separated process, and not running in the same JVM of the driver itself.
The release of EMB are linked in version number with the release of EvoMaster, as EvoMaster's libraries are used in the drivers (e.g., to clean databases and configure auth info).
In the Git repository of EMB, we did tag the versions of EMB.
See the releases page.
For example, to use version X
, you can check out the Git commit
of EMB tagged with version X
.
To see the current available tags, from a command-line you can execute:
git tag
Then, to switch to a specific tag X (e.g., v1.0.0
), you can run:
git checkout tags/v1.0.0
Finally, if for any reason you need to switch back to the latest snapshot version, you can run:
git checkout master
There is an issue if you try to checkout an old version.
Not only Java broke backward compatibility with JDK 9, but also Maven...
If you try to build with Maven and get an error regarding
maven-processor-plugin
, you might have to add manually
the following plugin dependency version:
<plugin>
<groupId>org.bsc.maven</groupId>
<artifactId>maven-processor-plugin</artifactId>
<version>3.3.3</version>
</plugin>
Branch develop is using the most recent SNAPSHOT version of EvoMaster. As that is not published online, you need to clone its repository, and build it locally (see its documentation on how to do it).