This repository contains example projects in Swift for iOS (using Tuist) to show how to generate a test coverage XML file for use with SonarCloud. It uses 4 different tools to generate those files so that they can be compared.
When I first looked into how to generate an XML file for SonarQube, I tried XCResultParser, and compared this with Slather, and saw very different results in terms of file size (XCResultParser was much bigger). I wanted to understand why they were so different, so I took a deep dive into these tools to get full understanding of what is going on, and rationalise on their differences.
The projects are a demonstrator for generating test code coverage data using 4 different methods
- 1: SonarQube script (shell script)
- 2: XCResultParser (Swift)
- 3: Xcode Coverage Generator (Swift)
- 4: Slather (Ruby)
The first 3 tools are all based on xccov command line tool (part of Xcode tools) that extract data from an XCResult bundle, and output XML to the terminal.
Slather is based on llvm-cov and extracts coverage data from a Coverage.profdata binary file.
Slather calls xcodebuild -showBuildSettings on the project file to find the path for derived data. In Tuist generated projects this doesn't work because the identifier doesn't match the folder name in derived data, so here I have chosen to use a YAML configuration file (.slather.yml) instead, where I can specify the folder in one of the properties. To determine the correct folder a script (generate_slather_yml.sh) checks the default derived data folder and finds the latest updated folder with the name of the project, and extracts out the identifier and updates the .slather.yml in the project. To generate the XML file, all that is required is to execute "slather" in the folder of the project.
The Slather YAML has a property for sources that functions as a filter for the coverage data. A point to note is that it doesn't use the source code - only the file name is used as a filter.
In addition to that, an advantage that Slather has over the other options is that files can be excluded by name. This improves the overall coverage accuracy for this case as I don't care about the source files generated by Tuist.
This repo uses the following tools:
- Tuist.io to generate the Xcode project and workspace.
It uses the following parsing tools:
- XCResultParser
- XcodeCoverageConverter (xcc)
- Slather
This installs all the tools.
This script calls the 4 run scripts that first clear any test output, and then calls "tuist test".
It requires Xcode 15.1, the iPhone 15, and iOS 17.2 to run. You may have to adjust it for the Xcode and iOS versions you have available.
The test command will generate an XCResult bundle in each project in the output folder.
This script uses the slather templates and searches the default Xcode derived data folder (~/Library/Developer/Xcode/DerivedData) to find the path for the Coverage.profdata.
This script runs all 4 parsing commands on all 4 projects and places everything in their respective output folders.
xccov_to_sonarqube_generic.sh is the script copied from SonarQube on Github.
XCResultParser supports more formats than just SonarQube XML, such as Cobertura XML and JUnit XML, and the scripts are there with parameters to experiment.
Although I expected the compiled code with Swift to be faster than a shell script, there was no noticeable difference in a large project with over 200k LOC. Even with larger projects that have > 7k tests, options 1-3 all ran at a similar throughput. The execution time was a few seconds on Circle CI. Slather was faster, taking less than 2 seconds.
For the options 1-3 that are based on xcccv, it is important to set the coverage to the targets of interest, as the default option will add all the testing code and external dependencies to the coverage. For the 4th option (Slather) selecting the testing targets makes no difference, as the binary output (Coverage.profdata) is the same irrespective of values set.
Since the SonarQube script is the simplest choice here, I would use that over options 2 and 3. Slather can be used as a slightly faster option, but requires more effort to configure it.