ICST-2022-submission: JavaScript Instrumentation for Search-Based Software Testing: A Study with RESTful APIs
In this package, we provide necessary information for replicating the experiment in the paper that includes
- EvoMaster: tools described in the paper with our js instrumentation and js test writer.
- jar: runnable jar for EvoMaster.
- EMB-js: five NodeJS case studies employed for the experiment.
- build.py: a python script to build tools and all five case studies.
- run.sh and runBB.sh: examples to run our tool on ncs-js case study.
- Results: raw data.
- r-script: a R script to analyze results and generate table and figures reported in the paper.
To enable SBST for NodeJs application, we implemented
- javascript instrumentation as Babel plugin, and the source code is located
Evomaster\client-js\evomaster-client-js
here. - driver to communicate with EvoMaster and SUT.
- JavaScript test writer for producing jest tests.
Step 1. In this repo, we provide a python script to build tools and all five case studies.
Go to the root, run
python3 build.py
After the build is finished, for each of case studies, there would be a new build
folder which contains instrumented code.
For EvoMaster, its jar
file would be found under EvoMaster/core/target
.
Step 2. To try our tool with ncs-js case study, run bash
./run.sh
The script includes two settings i.e., JS-MIO and JS-Random as described in the paper.
After the execution is done, you will see a folder named example
which contains
EM_JS_MIO_Test.js
: a test generated by JS-MIO with fitness evaluation on 100,000 HTTP callsEM_JS_Random_Test.js
: a test generated by JS-Random with fitness evaluation on 100,000 HTTP callsstatistics_ncs_js.csv
: statistics info (like covered targets, covered lines, covered branches) for two settingssnapshot_ncs_js.csv
: snapshot of statistics collected (at e.g., every 5% budget used) during the search for two settings
Step 3. In addition, we also provide an example for presenting coverage collection with c8
for a whole process.
The example is with BB_ft setting on ncs-js case study.
NOTE Before running the example, please install c8
, see https://github.com/bcoe/c8. Then start the example with
./runBB.sh
After the execution is done, you will see
In example
folder
EM_BB_ft_Test.js
is generated for a test generated by BB_ft setting with 3.97minutes time budgetstatistics_ncs_js.csv
: there are additional info for BB_ft settingsnapshot_ncs_js.csv
: there are additional info for BB_ft setting
In EMB-js/rest/ncs
, there would be a new folder named coverage
which contains
coverage-summary.json
: a coverage report with json format.
Raw data in our experiment are provided in results
folder
compressedData.zip
andsnapshotCompressedData.zip
are compressed statistics and snapshot files for white-box experiment;wb-exp
contains coverage reports based on generated tests;bb-exp
contains coverage reports which are collected during a whole process as therunBB.sh
example.
We also provide a R script js-exp.R to analyze the result and produce tables and figures used in the paper.
After run js-exp.R
, in results/generatedfiles
,
table_wbtargets_all.tex
is Table II for RQ1;- for RQ1, we also provide plot-lines based on the number of covered targets over the course of the search for each of cases studies, i.e.,
plot_wbtargets_cyclotron-js.pdf
(cyclotron-js),plot_wbtargets_disease-sh-api-js.pdf
(disease-sh-api),plot_wbtargets_ncs-js.pdf
(ncs-js),plot_wbtargets_nestjs-realworld-example-app-js.pdf
(realworld-app-js), andplot_wbtargets_scs-js.pdf
(scs-js); table_avg_coverage.tex
is Table III for RQ2 and RQ3;table_avg_faults.tex
is Table IV for RQ2 and RQ3;table_pair.tex
is Table V for RQ3.
Build JavaScript Instrumentation
Go to Evomaster\client-js\evomaster-client-js
, run
- step 1
npm install
- step 2
npm run build
Build EvoMaster
Go to Evomaster
, run
- step 3
mvn clean install -DskipTests
Instrument SUT
Go to a sut, e.g., EMB-js\rest\ncs-js
, run
- step 4
npm install
- step 5
npm run build
then under the sut, you will see a new folder named build
which contains instrumented code,
e.g., EMB-js\rest\ncs-js\build
Start EM Driver
Go to the sut, e.g., EMB-js\rest\ncs-js
- step 6
npm run em
Start EvoMaster
- step 7
java -jar evomaster.jar
In this experiment, we use configurations like,
--algorithm=MIO
for algorithm setting;--outputFormat=JS_JEST
for output format setting;--stoppingCriterion=FITNESS_EVALUATIONS --maxActionEvaluations=100000
for fitness budget setting;--maxTime
for time budget setting;--blackbox
for white-box and black-box setting.--bbSwaggerUrl
for accessing schema of REST in black-box setting.