chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase #1482

mpeddada1 · 2023-03-13T15:13:56Z

Follow up to #1430

Notable Changes

Set sonar.coverage.jacoco.xmlReportPaths to the jacoco xml file created in coverage-report.
Add sonar.skip to the modules that need to be excluded from showing up in the showcase reporting projects on Sonarcloud. These include java-iam, gapic-generator-java, etc.

Sample illustration of results

The result will show up in the following format on the main branch where a separate project has been created for each type of analysis (one for showcase unit test analysis and another for showcase integration test analysis).
The images below are provided for illustrative purposes and display results from an experiment conducted under a test Organization for a forked gapic-generator-java repo (https://sonarcloud.io/organizations/mpeddada1/projects?sort=-name)

Example of showcase_integration_tests coverage from tests with forked repo. :

Note of caution when comparing Jacoco vs SonarCloud: There is a slight discrepancy between the %s reported in SonarCloud vs Jacoco which we've used to gather the coverage data. For example, for integration test reporting, while the jacoco report in #1430 shows 15% branch coverage, SonarCloud in the test conducted shows 17.5% for conditional coverage. This could be attributed to how each of the tools interpret and calculate each coverage type. Reference: https://community.sonarsource.com/t/sonarqube-and-code-coverage/4725#h-1-to-1-comparison-between-sonarqube-and-coverage-report-is-not-relevant-3 and https://stackoverflow.com/a/40342671/14357112

…r-java into sonar-coverage

mpeddada1 · 2023-03-13T16:46:32Z

Error:  Error resolving version for plugin ' org.sonarsource.scanner.maven:sonar-maven-plugin' from the repositories [local (/home/runner/.m2/repository), central (https://repo.maven.apache.org/maven2)]: Plugin not found in any plugin repository -> [Help 1]

mpeddada1 · 2023-03-13T20:42:53Z

Error:  Errors: 
Error:    ITFirstHttp.testEcho » Unknown java.net.ConnectException: Connection refused (Connection refused)
Error:    ITFirstRpc.testEcho » Unavailable io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Error:    ITNumericEnums.verifyEnums » Unknown java.net.ConnectException: Connection refused (Connection refused)
[INFO] 
Error:  Tests run: 4, Failures: 0, Errors: 3, Skipped: 0
[INFO]

Showcase integration tests expect the showcase server to be running

blakeli0 · 2023-03-20T21:47:50Z

.github/workflows/sonar.yaml

@@ -1,4 +1,6 @@
 name: SonarCloud Build
+env:


We have showcase version here and in ci-maven.yaml as well, I'm not very familiar with Github actions, is it possible to configure a shared SHOWCASE_VERSION?

I think it is! According to the docs, shared variables can be created by doing Settings > secrets and variables > Actions > Manage environments. However, I'm not able to see the Settings tab for this repo so I think it requires some special permissions?

If we are going to have hermetic builds, the version needs to be stored in source.

Got it, thank you. Managing the variables in Settings may not be the best solution in that case. I'm inclined to keep the duplicate showcase_versions until an alternate solution becomes available in favor of making the builds self-contained and maintaining visibility into the showcase version we are testing against. Open to hearing more thoughts on this though.

blakeli0 · 2023-03-20T21:49:21Z

api-common-java/pom.xml

@@ -131,4 +131,19 @@
            </plugin>
        </plugins>
    </build>
+
+    <!-- Skip api-common when analyzing showcase test coverage on SonarCloud   -->


Why do we skip api-common? It's also a handwritten runtime dependency of client libraries, same as gax.

Currently the test coverage of showcase is restricted to the showcase and gax modules because of how heavily dependent GAPIC clients are on gax (which contains a lot of the transport logic). In terms of development, a feature or a bug fix oftentimes has a corresponding change in gax.

However, an argument can be made in support of api-common being included. The coverage report for showcase is to provide insights into how much of the core code the showcase tests are exercising, observe coverage drop/rise as we continue developing and also help us determine if it is safe to replace the existing handwritten integration tests. Given this purpose, do we think having a combined test coverage of api-common, gax-java and showcase tells a more complete story?

cc/ @burkedavison

Makes sense to me. If it's part of the 'platform runtime', let's include it. In the future this may also include java-core if we have a handwritten layer on top of showcase.

Sounds good! Will look into adding this in as well.

Done, updated the README and PR description as well.

…r-java into sonar-coverage

sonarcloud · 2023-03-21T21:10:12Z

[gapic-generator-java-root] Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

sonarcloud · 2023-03-21T21:13:33Z

[java_showcase_integration_tests] Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

sonarcloud · 2023-03-21T21:17:04Z

[java_showcase_unit_tests] Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

mpeddada1 added 16 commits March 2, 2023 15:59

chore: add aggregate test coverage collection for showcase and gax

d7c919a

add profile to collect test coverage

135022c

add fmt-maven-plugin

d2be02f

add maven compiler version property

22cda89

skip fmt

cd0de15

add version to fmt plugin

cddb9f2

add jar plugin to showcase

c116da1

add jar plugin to gapic-showcase

208cfff

empty commit to rerun builds

02ef0cb

empty commit

7f720b8

fix syntax in dep declaration

68557fe

remove spaces

57ddf93

add README for coverage-report module

fcc5257

replace monorepo with repsitory in README

45a0987

add profile to specify modules

47cf349

chore: add sonarcloud reporting for showcase test coverage

e61d6ea

product-auto-label bot added the size: l Pull request size is large. label Mar 13, 2023

mpeddada1 marked this pull request as ready for review March 13, 2023 15:32

mpeddada1 requested a review from a team as a code owner March 13, 2023 15:32

mpeddada1 and others added 4 commits March 13, 2023 11:35

Merge branch 'main' into sonar-coverage

83f085d

fix yaml syntax

d4ba320

Merge branch 'sonar-coverage' of github.com:googleapis/gapic-generato…

b856e7f

…r-java into sonar-coverage

fix spacing

f574ec8

mpeddada1 added 4 commits March 13, 2023 16:21

comment out previous step

61bb283

avoid calling clean in mvn command

6c54fbe

modify syntax:

644edc8

omit yaml formatting

afcf7b1

install and run showcase server

cff1fa4

mpeddada1 added 4 commits March 14, 2023 16:38

add clean

efc6ecc

add unit test step

9d676c0

fix syntax

03fd1b9

fix syntax

74fbf20

mpeddada1 mentioned this pull request Mar 15, 2023

chore: add aggregate test coverage collection of gax within the showcase and gax modules #1430

Merged

rebase with main

7e4202a

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Mar 16, 2023

clean up pom.xml

6264511

mpeddada1 requested review from burkedavison, blakeli0 and meltsufin March 16, 2023 18:25

burkedavison approved these changes Mar 17, 2023

View reviewed changes

blakeli0 reviewed Mar 20, 2023

View reviewed changes

mpeddada1 and others added 3 commits March 21, 2023 12:18

add api-common coverage

5a662a1

fix pom syntax

1ab860b

update screenshots in README

bcd8299

mpeddada1 changed the title ~~chore: add sonarcloud reporting for Showcase-GAX test coverage~~ chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase Mar 21, 2023

mpeddada1 requested review from burkedavison and blakeli0 March 21, 2023 18:20

burkedavison approved these changes Mar 21, 2023

View reviewed changes

mpeddada1 added 2 commits March 21, 2023 17:04

fix renovate version comments

27afb2e

Merge branch 'sonar-coverage' of github.com:googleapis/gapic-generato…

4b60a61

…r-java into sonar-coverage

blakeli0 approved these changes Mar 22, 2023

View reviewed changes

mpeddada1 merged commit c3d629c into main Mar 22, 2023

mpeddada1 deleted the sonar-coverage branch March 22, 2023 19:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase #1482

chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase #1482

mpeddada1 commented Mar 13, 2023 •

edited

Loading

mpeddada1 commented Mar 13, 2023

mpeddada1 commented Mar 13, 2023

blakeli0 Mar 20, 2023 •

edited

Loading

mpeddada1 Mar 21, 2023

burkedavison Mar 21, 2023

mpeddada1 Mar 21, 2023

blakeli0 Mar 20, 2023

mpeddada1 Mar 21, 2023

burkedavison Mar 21, 2023

mpeddada1 Mar 21, 2023

mpeddada1 Mar 21, 2023

sonarcloud bot commented Mar 21, 2023

sonarcloud bot commented Mar 21, 2023

sonarcloud bot commented Mar 21, 2023

chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase #1482

chore: add sonarcloud reporting for aggregate test coverage in gax, api-common and showcase #1482

Conversation

mpeddada1 commented Mar 13, 2023 • edited Loading

mpeddada1 commented Mar 13, 2023

mpeddada1 commented Mar 13, 2023

blakeli0 Mar 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Mar 21, 2023

sonarcloud bot commented Mar 21, 2023

sonarcloud bot commented Mar 21, 2023

mpeddada1 commented Mar 13, 2023 •

edited

Loading

blakeli0 Mar 20, 2023 •

edited

Loading