From ad47e7d5a6e63487a15383bba439b715f8a7c5a2 Mon Sep 17 00:00:00 2001 From: "W. Trevor King" Date: Thu, 30 Nov 2017 10:12:52 -0800 Subject: [PATCH] Makefile: Change from prove to node-tap For better reporting, including skips. Prove (at least as of TAP::Harness v3.35_01 and Perl v5.22.3) seems to report skips only when there were also failures. For example, here's a skip-only test: $ validation/mounts.t TAP version 13 1..1 ok 1 # SKIP TODO: mounts generation options have not been implemented $ prove validation/mounts.t validation/mounts.t .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU) Result: PASS node-tap (as of version 11.0.0) reports that skip: $ tap validation/mounts.t validation/mounts.t ................................... 0/1 Skipped: 1 TODO: mounts generation options have not been implemented total ................................................. 0/1 0 passing (27.297ms) 1 pending And here's a skip with a failure test: $ ./test-skip TAP version 13 1..2 not ok 1 - failing ok 2 # SKIP: skipping Prove warns about the skip now: $ prove test-skip test-skip .. Failed 1/2 subtests (less 1 skipped subtest: 0 okay) Test Summary Report ------------------- test-skip (Wstat: 0 Tests: 2 Failed: 1) Failed test: 1 Files=1, Tests=2, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL But node-tap has a nicer warning: $ tap ./test-skip ./test-skip ........................................... 0/2 not ok failing Skipped: 1 : skipping total ................................................. 0/2 0 passing (39.088ms) 1 pending 1 failing Similarly, node-tap does a better job handling YAML blocks [1]: $ ./test-yaml TAP version 13 1..4 ok 1 - success diagnostic # success not ok 2 - failure diagnostic # failure ok 3 - success YAML --- message: success ... not ok 4 - failure YAML --- message: failure ... Prove either shows no diagnostics: $ prove test-yaml test-yaml .. Failed 2/4 subtests Test Summary Report ------------------- test-yaml (Wstat: 0 Tests: 4 Failed: 2) Failed tests: 2, 4 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL or it shows all the diagnostics (even for successful tests): $ prove --comments test-yaml test-yaml .. 1/? # success # failure test-yaml .. Failed 2/4 subtests Test Summary Report ------------------- test-yaml (Wstat: 0 Tests: 4 Failed: 2) Failed tests: 2, 4 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL I can't find a way to get Prove to show the YAML blocks. node-tap, on the other hand, does the right thing with YAML blocks: $ tap ./test-yaml ./test-yaml ........................................... 2/4 not ok failure diagnostic not ok failure YAML message: failure total ................................................. 2/4 2 passing (42.409ms) 2 failing We don't use YAML blocks at the moment, but I'm going to transition to them later. [1]: http://testanything.org/tap-version-13-specification.html#yaml-blocks Signed-off-by: W. Trevor King --- Makefile | 3 +- README.md | 165 +++++++++++++++++++++++++++++++++++------------------- 2 files changed, 108 insertions(+), 60 deletions(-) diff --git a/Makefile b/Makefile index 071e54037..a7a858123 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,6 @@ PREFIX ?= $(DESTDIR)/usr BINDIR ?= $(DESTDIR)/usr/bin +TAP ?= tap BUILDTAGS= RUNTIME ?= runc @@ -39,7 +40,7 @@ clean: rm -f oci-runtime-tool runtimetest *.1 $(VALIDATION_TESTS) localvalidation: - RUNTIME=$(RUNTIME) prove $(VALIDATION_TESTS) + RUNTIME=$(RUNTIME) $(TAP) $(VALIDATION_TESTS) .PHONY: validation-executables validation-executables: $(VALIDATION_TESTS) diff --git a/README.md b/README.md index f987a6b14..5f16436bc 100644 --- a/README.md +++ b/README.md @@ -30,72 +30,84 @@ INFO[0000] Bundle validation succeeded. ## Testing OCI runtimes -The runtime validation suite uses [`prove`][prove], which is packaged for most distributions (for example, it is in [Debian's `perl` package][debian-perl] and [Gentoo's `dev-lang/perl` package][gentoo-perl]). -If you cannot install `prove`, you can probably run the test suite with another [TAP consumer][tap-consumers], although you'll have to edit the [`Makefile`](Makefile) to replace `prove`. +The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]). +If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it: + +```console +$ npm install tap +``` ```console $ make runtimetest validation-executables -$ sudo make RUNTIME=runc localvalidation -RUNTIME=runc prove validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t -validation/linux_rootfs_propagation_shared.t ...... Failed 1/19 subtests -validation/create.t ............................... ok -validation/default.t .............................. ok -validation/linux_readonly_paths.t ................. ok -validation/linux_masked_paths.t ................... Failed 1/19 subtests -validation/mounts.t ............................... ok -validation/process.t .............................. ok -validation/root_readonly_false.t .................. ok -validation/linux_sysctl.t ......................... ok -validation/linux_devices.t ........................ ok -validation/linux_gid_mappings.t ................... Failed 1/19 subtests -validation/process_oom_score_adj.t ................ ok -validation/process_capabilities.t ................. ok -validation/process_rlimits.t ...................... ok -validation/root_readonly_true.t ................... ok -validation/linux_rootfs_propagation_unbindable.t .. failed to create the container +RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t +validation/linux_rootfs_propagation_shared.t ........ 18/19 + not ok rootfs propagation + +validation/create.t ................................... 4/4 +validation/default.t ................................ 19/19 +validation/linux_readonly_paths.t ................... 19/19 +validation/linux_masked_paths.t ..................... 18/19 + not ok masked paths + +validation/mounts.t ................................... 0/1 + Skipped: 1 + TODO: mounts generation options have not been implemented + +validation/process.t ................................ 19/19 +validation/root_readonly_false.t .................... 19/19 +validation/linux_sysctl.t ........................... 19/19 +validation/linux_devices.t .......................... 19/19 +validation/linux_gid_mappings.t ..................... 18/19 + not ok gid mappings + +validation/process_oom_score_adj.t .................. 19/19 +validation/process_capabilities.t ................... 19/19 +validation/process_rlimits.t ........................ 19/19 +validation/root_readonly_true.t ...................failed to create the container rootfsPropagation=unbindable is not supported exit status 1 -validation/linux_rootfs_propagation_unbindable.t .. Dubious, test returned 1 (wstat 256, 0x100) -No subtests run -validation/hostname.t ............................. ok -validation/linux_uid_mappings.t ................... failed to create the container +validation/root_readonly_true.t ..................... 19/19 +validation/linux_rootfs_propagation_unbindable.t ...... 0/1 + not ok validation/linux_rootfs_propagation_unbindable.t + timeout: 30000 + file: validation/linux_rootfs_propagation_unbindable.t + command: validation/linux_rootfs_propagation_unbindable.t + args: [] + stdio: + - 0 + - pipe + - 2 + cwd: /…/go/src/github.com/opencontainers/runtime-tools + exitCode: 1 + +validation/hostname.t ...................failed to create the container User namespace mappings specified, but USER namespace isn't enabled in the config exit status 1 -validation/linux_uid_mappings.t ................... Dubious, test returned 1 (wstat 256, 0x100) -No subtests run - -Test Summary Report -------------------- -validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 16 -validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 13 -validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1) - Failed test: 19 -validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0) - Non-zero exit status: 1 - Parse errors: No plan found in TAP output -validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0) - Non-zero exit status: 1 - Parse errors: No plan found in TAP output -Files=18, Tests=271, 31 wallclock secs ( 0.06 usr 0.01 sys + 0.52 cusr 0.21 csys = 0.80 CPU) -Result: FAIL -make: *** [Makefile:42: localvalidation] Error 1 +validation/hostname.t ............................... 19/19 +validation/linux_uid_mappings.t ....................... 0/1 + not ok validation/linux_uid_mappings.t + timeout: 30000 + file: validation/linux_uid_mappings.t + command: validation/linux_uid_mappings.t + args: [] + stdio: + - 0 + - pipe + - 2 + cwd: /…/go/src/github.com/opencontainers/runtime-tools + exitCode: 1 + +total ............................................. 267/273 + + + 267 passing (31s) + 1 pending + 5 failing + +make: *** [Makefile:43: localvalidation] Error 1 ``` -If you are confident that the `validation/*.t` are current, you can run `prove` (or your preferred TAP consumer) directly to avoid unnecessary rebuilds: - -```console -$ sudo RUNTIME=runc prove -Q validation/*.t - -Test Summary Report -------------------- -… -Files=18, Tests=271, 31 wallclock secs ( 0.07 usr 0.01 sys + 0.51 cusr 0.22 csys = 0.81 CPU) -Result: FAIL -``` - -And you can run an individual test executable directly: +You can also run an individual test executable directly: ```console $ RUNTIME=runc validation/default.t @@ -122,10 +134,45 @@ ok 19 - gid mappings 1..19 ``` +If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers]. +For example, with [`prove`][prove]: + +```console +$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation +RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t +failed to create the container +rootfsPropagation=unbindable is not supported +exit status 1 +failed to create the container +User namespace mappings specified, but USER namespace isn't enabled in the config +exit status 1 + +Test Summary Report +------------------- +validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 16 +validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 13 +validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0) + Non-zero exit status: 1 + Parse errors: No plan found in TAP output +validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0) + Non-zero exit status: 1 + Parse errors: No plan found in TAP output +validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1) + Failed test: 19 +Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU) +Result: FAIL +make: *** [Makefile:43: localvalidation] Error 1 +``` + [bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md [config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md -[debian-perl]: https://packages.debian.org/stretch/perl -[gentoo-perl]: https://packages.gentoo.org/packages/dev-lang/perl +[debian-node-tap]: https://packages.debian.org/stretch/node-tap +[debian-nodejs]: https://packages.debian.org/stretch/nodejs +[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs +[node-tap]: http://www.node-tap.org/ +[npm]: https://www.npmjs.com/ [prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove [runC]: https://github.com/opencontainers/runc [runtime-spec]: https://github.com/opencontainers/runtime-spec