Set up code coverage reports on GitHub Actions

[MaxDesiatov]

I think stability of our existing code should be of higher priority. We need to be diligent about keeping test coverage high and not reducing it significantly (ideally increasing it, as it's probably already too low) in any PRs. GitHub Actions job that runs in a PR and posts current coverage as a comment would be super helpful.

After that is done we can set up more complex workflows that compute coverage deltas. I know that proprietary solutions like Codecov can already do this, but I'm wary of relying on them as they tend to shut down free access for open-source projects after some period of initial unsustainable growth. GitHub Actions seems to be a more reliable tool for this long term.

[mattpolzin]

In the “easy to integrate, easy to move on from” category, I can say my codecov action has been good enough for me so far (and obviously improvements are welcome!).

https://github.com/mattpolzin/swift-codecov-action

here it is in action; it currently just prints a coverage table, optionally fails if coverage standards are not met, and results in a CI checkmark on PRs.
https://github.com/mattpolzin/OpenAPIKit/runs/1623070311?check_suite_focus=true

[MaxDesiatov]

Lovely! I wonder how hard it would be to modify it to post coverage results as comments? We previously had some success in creating GitHub releases from shell scripts directly using curl, maybe that could be done in your action too?

[mattpolzin]

One option would be to “fix” the current limitations in what my action outputs and pair it with other GitHub actions (I’m sure there’s one out there that does nothing but post a comment).

[mattpolzin]

(I’m not opposed to adding features like that to my action, I just generally find GitHub actions to be so conveniently composable)

[carson-katri]

Is it possible to put it in the diff like the linter comments? That way it doesn't clutter the discussion.

[mattpolzin]

Definitely (assuming I catch your drift). Creating a file and committing it is not so different effort-wise to creating a new PR comment from a GitHub action.

[MaxDesiatov]

Our SwiftLint warnings are attached in diffs to specific lines, while code coverage percentage is global per project.

I like the idea of committing that percentage as a file though, that way it's simpler to compare coverage of a PR branch with coverage in the main branch. We wouldn't have to run tests again to calculate the value for the main branch then. We could just check out the main branch in the job and load the value from the file for comparison.

[mattpolzin]

Bummer, using swift-codecov-action for this is easier said than done (I am only speaking to one possible way to approach this ticket, of course). swift-codecov-action runs in a docker container which means it needs to run in an ubuntu job rather than a macOS job. Currently, that means this project cannot be tested because of the TokamakDemo product. That product is not even needed by the test target, but swift test (unlike swift build) refuses to build a subset of all targets... (best I have been able to figure)

[MaxDesiatov]

We already work around that by building just the test product with swift build --product TokamakPackageTests and running the test bundle manually with xctest here

Tokamak/.github/workflows/ci.yml

Line 29 in b55f703

swift build --product TokamakPackageTests

[mattpolzin]

We already work around that...

Right, but that runs-on MacOS where xctest is available whereas the test coverage action I was trying to use runs in an Ubuntu Docker image; GitHub does not currently offer Docker support for MacOS build boxes (last I checked).

It would not be so terrible to pull the swift-test-codecov tool that the action currently uses in directly and just build and run it in a MacOS environment in order to get past this, but it’s a different direction than I started taking initially.

[MaxDesiatov]

On Ubuntu you'd do essentially the same thing, build just the tests with swift build --product TokamakPackageTests, then run the test bundle. You won't need xctest though, the test bundle is just a plain executable on Linux.

[MaxDesiatov]

The test bundle path would be the same I guess, .build/debug/TokamakPackageTests.xctest, or .build/debug/TokamakPackageTests, I don't remember which one exactly.

[mattpolzin]

Oh, that’s cool. TIL!

[mattpolzin]

I wonder if it’s possible to generate the test coverage file this way though. Worth exploring. (Referring to the file created by —enable-test-coverage)

[mattpolzin]

Got that figured out; building with the right flags and telling swift test to skip building works.

Proof of concept with a rudimentary comment here: mattpolzin#2 / https://github.com/mattpolzin/Tokamak/pull/2/checks?check_run_id=1724996327

Requires an access token to be added to this repository before it will work (for the commenting).

[MaxDesiatov]

Perfect! 👏
Please feel free to submit a PR to this repo, I'll fix the token issue. Or maybe we'll find a way to attach the report to the PR diff, like our current SwiftLint action does, that wouldn't even require a token.

[mattpolzin]

I've been puttering away at this a bit (modifying CI can be such a drag) -- this comment is not-so-bad, I think (image attached). I removed all of the tests for Color to test the diff with main.

It really is just a diff, so not super great, but maybe better than nothing? I'll PR for discussion.