NG run-testplans action

[thomaseizinger]

Currently, this is a workflow but it could just be a composite action which would remove the artifact up and downloading.

Relevant discussion: libp2p/rust-libp2p#3331 (comment)

Suggestions on new API design for the action/test-runner: libp2p/rust-libp2p#3331 (comment)

In particular:

• Change API to accept a series of JSON files
• Each file describes capabilities of one test binary (i.e. one JSON object, no arrays)

Happy to implement these changes once we agree on them :)

[MarcoPolo]

Change API to accept a series of JSON files

Yup. like npm test -- --extra-version=./foo.json --extra-version=./bar.json works for me.

Each file describes capabilities of one test binary (i.e. one JSON object, no arrays)

sgtm!

Let me know if you do end up picking this up. I'll do the same if I start on this today :)

[thomaseizinger]

Change API to accept a series of JSON files

Yup. like npm test -- --extra-version=./foo.json --extra-version=./bar.json works for me.

I was actually thinking something like:

npm test -- ./go-libp2p-v0.22.json ./go-libp2p-head.json ./rust-libp2p-head.json

In other words, arguments the same way mv takes a list of files to move.

The name of the files doesn't matter. I'd also consider getting rid of versions.ts and require the JSON files imported there to be passed on the commandline too.

Each file describes capabilities of one test binary (i.e. one JSON object, no arrays)

sgtm!

Let me know if you do end up picking this up. I'll do the same if I start on this today :)

Will do!

[MarcoPolo]

I'd also consider getting rid of versions.ts and require the JSON files imported there to be passed on the commandline too.

Getting type checking is very nice. I don't know if I want to give that up. I think of "version.ts" as the source of truth for what supported versions are capable of.

[thomaseizinger]

I'd also consider getting rid of versions.ts and require the JSON files imported there to be passed on the commandline too.

Getting type checking is very nice. I don't know if I want to give that up. I think of "version.ts" as the source of truth for what supported versions are capable of.

I just figured that if we pass the "extra" versions on the command-line we might as well pass all JSON files there to simplify the overall design. It seemed like a natural extension but you are right that we'd be losing type-checks on those files. Why are they declared in JSON files to begin with and not just within the array that is defined in versions.ts?

[MarcoPolo]

Why are they declared in JSON files to begin with and not just within the array that is defined in versions.ts?

They are defined in the array? https://github.com/libp2p/test-plans/blob/master/multidim-interop/versions.ts#L18

The JSON imported is simply the imageID. JSON format so that we can import it instead of dealing with reading files.

cat js/v0.41/image.json 
{"imageID": "sha256:6ee28dd35f674362bbfd059b16abb9954a09eb2da67e9a8a2fbd0ffc4c7d3f32"}

[thomaseizinger]

Why are they declared in JSON files to begin with and not just within the array that is defined in versions.ts?

They are defined in the array? master/multidim-interop/versions.ts#L18

The JSON imported is simply the imageID. JSON format so that we can import it instead of dealing with reading files.

cat js/v0.41/image.json 
{"imageID": "sha256:6ee28dd35f674362bbfd059b16abb9954a09eb2da67e9a8a2fbd0ffc4c7d3f32"}

Ah yes, I understand what is happening now, sorry. My head was already in "we are pulling docker images"-land so I forgot that we are generating these JSON files at build time and thus need to import them. For rust-libp2p, there won't be any image.json files because we will be referencing the docker container ID directly in versions.ts. I think we are on the same page:

• Leave versions.ts as is
• Extend test runner to accept a list of JSON files that extend this array

[thomaseizinger]

Motivated by #121, here is another idea:

With the new action being a composite one, I think we can get rid of the separate checkout step if we bundle (using webpack, esbuild, etc) the JavaScript up into a separate JS GitHub action. For this to work, we'd have to make another, repository local JS action (under .github/actions) that we can reference by path from the composite one. That way, the hash of the action we reference in the other repositories will always point to a specific version of the runner.

For it to be completely stateless, we would have to migrate the Go and JS test binaries to publish containers for their test binaries to a registry, the same way we do it for the rust-libp2p: https://github.com/libp2p/rust-libp2p/blob/master/.github/workflows/publish-interop-test-images.yml

Happy to invest some time into this if we agree on the design.

[thomaseizinger]

For it to be completely stateless, we would have to migrate the Go and JS test binaries to publish containers for their test binaries to a registry, the same way we do it for the rust-libp2p: libp2p/rust-libp2p@master/.github/workflows/publish-interop-test-images.yml

Doing so would shorten the runtime by ~10minutes as we don't have to build any more images: https://github.com/libp2p/test-plans/actions/runs/3998997158/jobs/6862340373

[thomaseizinger]

I've started to work on the composite action.

[MarcoPolo]

I’m not convinced publishing to a registry is better than a large cache. I would want to investigate #113 (comment) before we require implementations to publish to a registry.

[thomaseizinger]

I’m not convinced publishing to a registry is better than a large cache. I would want to investigate #113 (comment) before we require implementations to publish to a registry.

I managed to implement the "use entire repository at the specified revision without a dedicated checkout" without this requirement so we can optimise the performance either way now!

[thomaseizinger]

This can be considered done now that #123 is merged.