Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable coverage reporting #3100

Merged
merged 10 commits into from
May 20, 2021
+294 −479
Merged

Enable coverage reporting #3100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b82d545
chore: rip out per-package nyc support
michaelfig May 14, 2021
89ac4c7
feat: make top-level `yarn c8 ava` work
michaelfig May 14, 2021
d58f234
ci: build and deploy https://agoric-sdk-coverage.netlify.app
michaelfig May 14, 2021
d3b2549
fix: patch ESM to allow bottom-up implementation of type: "module"
michaelfig May 16, 2021
5877439
docs: describe limitations of c8 under source transforms
michaelfig May 16, 2021
93279c1
fix: convert swing-store-simple to "type": "module"
michaelfig May 16, 2021
6d1b9c7
ci: remove node 12.14.1 as it doesn't support "type":"module"
michaelfig May 16, 2021
3192156
ci: only generate coverage for NESM-supporting packages
michaelfig May 17, 2021
856abbe
refactor: update COVERAGE.md to accommodate ava-nesm.config.js
michaelfig May 18, 2021
464b464
ci: clean up to be ready for master
michaelfig May 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ag-solo-xs.yml.DISABLED
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- uses: actions/checkout@v2
- uses: actions/setup-node@v1
with:
node-version: '12.14.1'
node-version: '12.x'
- name: cache node modules
uses: actions/cache@v1
with:
Expand Down
66 changes: 59 additions & 7 deletions .github/workflows/test-all-packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -117,6 +117,58 @@ jobs:
name: benchmarkstats.json
path: packages/swingset-runner/benchstats*.json

coverage:
needs: build
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['14.x']
if: ${{github.event_name == 'push' && github.ref == 'refs/heads/master'}}
steps:
- uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
# BEGIN-RESTORE-BOILERPLATE
- name: restore built files
id: built
uses: actions/cache@v1
with:
path: .
key: ${{ runner.os }}-${{ matrix.node-version }}-built-${{ github.sha }}
- uses: actions/checkout@v2
with:
submodules: 'true'
if: steps.built.outputs.cache-hit != 'true'
- name: yarn install
run: yarn install
if: steps.built.outputs.cache-hit != 'true'
- name: yarn build
run: yarn build
if: steps.built.outputs.cache-hit != 'true'
# END-RESTORE-BOILERPLATE

- name: generate coverage for all tests
run: 'yarn test:c8-all || :'
- name: generate coverage/html reports
run: yarn c8 report --reporter=html-spa --reports-dir=coverage/html --temp-directory=coverage/tmp
- uses: actions/upload-artifact@v2
with:
name: coverage
path: coverage
- name: Find Netlify site ID
run: |
echo "NETLIFY_SITE_ID=$(cat COVERAGE_NETLIFY_SITE_ID)" >> $GITHUB_ENV

- uses: nwtgck/actions-netlify@v1.1
with:
# Production deployment if a commit to master.
production-deploy: ${{ github.ref == 'refs/heads/master' }}
publish-dir: coverage/html
# SECURITY: we don't want to hand out the Github token to this action.
# github-token: ${{ secrets.GITHUB_TOKEN }}
michaelfig marked this conversation as resolved.
Show resolved Hide resolved
env:
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}

##################
# Fast-running tests run as a group:
test-quick:
Expand All @@ -125,7 +177,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down Expand Up @@ -254,7 +306,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down Expand Up @@ -320,7 +372,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down Expand Up @@ -358,7 +410,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down Expand Up @@ -405,7 +457,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down Expand Up @@ -450,7 +502,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.14.1', '12.x', '14.x']
node-version: ['12.x', '14.x']
steps:
- uses: actions/setup-node@v1
with:
Expand Down
72 changes: 72 additions & 0 deletions COVERAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Coverage reports

## Caveat

Until each module can be migrated to support Node.js's builtin ESM
implementation (`nesm`), the coverage line numbers will be out-of-sync with
reality.

In addition, we will have to implement source maps in all of our
source-to-source transforms (such as `@agoric/bundle-source`,
`@agoric/transform-metering`, and `@agoric/static-module-record`).

## Reports

Coverage reports for the current main branch (whose packages support `nesm`) are
published by CI to: https://agoric-sdk-coverage.netlify.app

You can create a report in any package (including the top-level directory):

```sh
# Get options available for coverage:
yarn c8 --help
# Run ava under Node.js coverage and display a summary:
yarn c8 -a ava
# Generate a nice, detailed HTML report:
yarn c8 report --reporter=html-spa
open coverage/html/index.html
```

## Node.js ESM Support

With the current `patches/esm+3.2.25.diff`, it is possible to migrate packages
to support both resm (`-r esm`) and nesm (Node.js ESM Support). If an
`agoric-sdk` package has dependencies that support nesm, you can attempt to make
it also support nesm by:

1. Create `ava-nesm.config.js` removing `"require": ["esm"]`:

```sh
../../scripts/ava-nesm.cjs > ava-nesm.config.js
```

2. Make the following changes to its `package.json` (omitting comments):

```js
{
// Enable nesm support.
"type": "module",
"scripts": {
// The following line enables coverage generation from the top.
"test:c8": "c8 $C8_OPTIONS ava --config=ava-nesm.config.js"
}
}
```

3. Test that both `yarn test` and `yarn test:c8` run correctly.

## Planned Implementation

Our runtime source transforms can be conditional on the `$NODE_V8_COVERAGE`
environment variable, which is set by `c8`.

When that is nonempty, source transforms can implement special behaviour to
preserve source maps for transformed code. This involves using a `//#
sourceURL=...` tag to associate a unique URL to an actual existing file during
`eval`. The file's contents include a `//# sourceMappingURL=...`, whether a
relative URL (taken from `sourceURL`) to a `.js.map` file or inline as a `data:`
URI.


The `sourceMappingURL` source map must contain a `"sources"` property as a
relative or absolute URL to the input source file.
1 change: 1 addition & 0 deletions COVERAGE_NETLIFY_SITE_ID
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
91d78371-7395-4779-8367-0c4f7088297c
1 change: 0 additions & 1 deletion golang/cosmos/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
"test": "exit 0",
"build:gyp": "make compile-gyp",
"build:gyp-debug": "make compile-gyp GYP_DEBUG=--debug",
"test:nyc": "nyc ava",
"test:xs": "exit 0",
"build": "exit 0",
"lint-fix": "eslint --fix '**/*.js'",
Expand Down
14 changes: 12 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
"devDependencies": {
"@typescript-eslint/parser": "^4.18.0",
"ava": "^3.12.1",
"c8": "^7.7.2",
"eslint": "^7.23.0",
"eslint-config-airbnb-base": "^14.0.0",
"eslint-config-jessie": "^0.0.4",
Expand All @@ -58,7 +59,6 @@
"eslint-plugin-jsx-a11y": "^6.2.3",
"eslint-plugin-prettier": "^3.1.2",
"lerna": "^3.20.2",
"nyc": "^15.1.0",
"prettier": "^1.18.2",
"typescript": "^4.2.3"
},
Expand All @@ -75,13 +75,23 @@
"lint-fix": "yarn workspaces run lint-fix",
"lint-check": "yarn workspaces run lint-check",
"lint": "yarn workspaces run lint-check",
"test": "yarn workspaces run test",
"test": "ava",
"test:c8-all": "rm -rf coverage/tmp && C8_OPTIONS=\"--clean=false --temp-directory=$PWD/coverage/tmp\" lerna run test:c8",
"test:xs": "yarn workspaces run test:xs",
"build": "yarn workspaces run build",
"postinstall": "patch-package",
"patch-package": "patch-package",
"build-xs-worker": "cd packages/xs-vat-worker && yarn build:xs-lin"
},
"ava": {
"files": [
"packages/*/test/**/test-*.js"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this interfere with / override ava settings in individual package's yarn test?

It looks like this is meant for use by a top-level yarn test which runs tests in all packages in parallel. When I last looked into doing that, I got stuck on the different require options we needed for different sets of tests (in particular importing install-ses, or install-metering-and-ses). Did you find a solution for that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this interfere with / override ava settings in individual package's yarn test?

Only when run from the top level. If yarn test is run in the specific package, the package's settings will take effect.

It looks like this is meant for use by a top-level yarn test which runs tests in all packages in parallel. When I last looked into doing that, I got stuck on the different require options we needed for different sets of tests (in particular importing install-ses, or install-metering-and-ses). Did you find a solution for that?

Yes. It turns out we only use "require" for two modules: "esm" (which appears in basically every "ava" config), and an enzyme setup script (only used in packages/ui-components/package.json). So, the situation is not nearly as grim.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think @dtribble was once keen on using the AVA config to jam a -r @agoric/install-ses into e.g. all zoe tests, to avoid adding a line to the top of every test file, and that was going to push us to e.g. move all @agoric/install-metering-and-ses tests into one directory, and have two top-level AVA calls (with one set of -r options each). I think we talked him down from that :).

I've got nodeArguments: [ "--expose-gc" ] in the SwingSet package.json, to enable GC controls while testing. We may need to move that to the top-level package.json to make sure swingset tests work both from the top level and from packages/SwingSet (and also when e.g. Zoe does swingset-based tests, although if they're short enough, we can probably tolerate the inability to provoke GC).

],
"require": [
"esm"
],
"timeout": "30m"
},
"dependencies": {
"patch-package": "^6.2.2"
}
Expand Down
4 changes: 1 addition & 3 deletions packages/ERTP/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
"test:xs": "yarn test:xs-unit && yarn test:xs-worker",
"test:xs-unit": "ava-xs",
"test:xs-worker": "SWINGSET_WORKER_TYPE=xs-worker ava -c 2 test/swingsetTests/**/test-*.js",
"test:nyc": "nyc ava",
"pretty-fix": "prettier --write '**/*.js'",
"pretty-check": "prettier --check '**/*.js'",
"lint-fix": "yarn lint:eslint --fix && yarn lint:types",
Expand Down Expand Up @@ -57,8 +56,7 @@
"@agoric/install-ses": "^0.5.13",
"@agoric/swingset-vat": "^0.17.2",
"ava": "^3.12.1",
"esm": "^3.2.25",
"nyc": "^15.1.0"
"esm": "^3.2.25"
},
"files": [
"src",
Expand Down
3 changes: 1 addition & 2 deletions packages/SwingSet/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@
"devDependencies": {
"@agoric/install-metering-and-ses": "^0.2.13",
"@agoric/ses-ava": "^0.1.1",
"ava": "^3.12.1",
"nyc": "^15.1.0"
"ava": "^3.12.1"
},
"dependencies": {
"@agoric/assert": "^0.2.12",
Expand Down
5 changes: 0 additions & 5 deletions packages/agoric-cli/.ava-unit-test.config.js
View file
Open in desktop

This file was deleted.

8 changes: 6 additions & 2 deletions packages/agoric-cli/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
],
"scripts": {
"build": "exit 0",
"test": "ava --config .ava-unit-test.config.js",
"test": "ava",
"test:xs": "exit 0",
"integration-test": "ava --config .ava-integration-test.config.js",
"lint-check": "eslint '**/*.{js,jsx}'",
Expand All @@ -22,7 +22,6 @@
"devDependencies": {
"@agoric/swingset-vat": "^0.17.2",
"ava": "^3.12.1",
"nyc": "^15.1.0",
"tmp": "^0.1.0"
},
"dependencies": {
Expand Down Expand Up @@ -59,6 +58,11 @@
"@agoric"
]
},
"ava": {
"files": ["test/**/test-*.js"],
"require": ["esm"],
"timeout": "2m"
},
"prettier": {
"trailingComma": "all",
"singleQuote": true
Expand Down
3 changes: 1 addition & 2 deletions packages/assert/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,7 @@
"devDependencies": {
"@agoric/install-ses": "^0.5.13",
"ava": "^3.12.1",
"esm": "^3.2.25",
"nyc": "^15.1.0"
"esm": "^3.2.25"
},
"files": [
"src/",
Expand Down
4 changes: 1 addition & 3 deletions packages/bundle-source/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
"scripts": {
"build": "exit 0",
"test": "ava",
"test:nyc": "nyc ava",
"test:xs": "exit 0",
"lint-fix": "eslint --fix '**/*.js'",
"lint-check": "eslint '**/*.js'",
Expand All @@ -18,8 +17,7 @@
},
"devDependencies": {
"@agoric/install-ses": "^0.5.13",
"ava": "^3.12.1",
"nyc": "^15.1.0"
"ava": "^3.12.1"
},
"dependencies": {
"@agoric/compartment-mapper": "^0.2.4",
Expand Down
4 changes: 1 addition & 3 deletions packages/captp/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,13 @@
"scripts": {
"build": "exit 0",
"test": "ava",
"test:nyc": "nyc ava",
"test:xs": "exit 0",
"lint-fix": "eslint --fix '**/*.js'",
"lint-check": "eslint 'lib/*.js'"
},
"devDependencies": {
"@agoric/install-ses": "^0.5.13",
"ava": "^3.12.1",
"nyc": "^15.1.0"
"ava": "^3.12.1"
},
"dependencies": {
"@agoric/eventual-send": "^0.13.14",
Expand Down
4 changes: 1 addition & 3 deletions packages/cosmic-swingset/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
"scripts": {
"build": "exit 0",
"test": "ava",
"test:nyc": "nyc ava",
"test:xs": "exit 0",
"pretty-fix": "prettier --write '**/*.js'",
"pretty-check": "prettier --check '**/*.js'",
Expand Down Expand Up @@ -46,8 +45,7 @@
"esm": "^3.2.25"
},
"devDependencies": {
"ava": "^3.12.1",
"nyc": "^15.1.0"
"ava": "^3.12.1"
},
"eslintConfig": {
"extends": [
Expand Down
4 changes: 1 addition & 3 deletions packages/deploy-script-support/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
"scripts": {
"build": "exit 0",
"test": "ava",
"test:nyc": "nyc ava",
"test:xs": "exit 0",
"pretty-fix": "prettier --write '**/*.js'",
"pretty-check": "prettier --check '**/*.js'",
Expand Down Expand Up @@ -58,8 +57,7 @@
"@agoric/install-ses": "^0.5.13",
"@agoric/swingset-vat": "^0.17.2",
"ava": "^3.12.1",
"esm": "^3.2.25",
"nyc": "^15.1.0"
"esm": "^3.2.25"
},
"files": [
"src",
Expand Down
Loading