Makefile: Change from prove to node-tap

For better reporting, including skips.  Prove (at least as of
TAP::Harness v3.35_01 and Perl v5.22.3) seems to report skips only
when there were also failures.  For example, here's a skip-only test:

  $ validation/mounts.t
  TAP version 13
  1..1
  ok 1 # SKIP TODO: mounts generation options have not been implemented
  $ prove validation/mounts.t
  validation/mounts.t .. ok
  All tests successful.
  Files=1, Tests=1,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
  Result: PASS

node-tap (as of version 11.0.0) reports that skip:

  $ tap validation/mounts.t
  validation/mounts.t ................................... 0/1
    Skipped: 1
       TODO: mounts generation options have not been implemented

  total ................................................. 0/1

    0 passing (27.297ms)
    1 pending

And here's a skip with a failure test:

  $ ./test-skip
  TAP version 13
  1..2
  not ok 1 - failing
  ok 2 # SKIP: skipping

Prove warns about the skip now:

  $ prove test-skip
  test-skip .. Failed 1/2 subtests
          (less 1 skipped subtest: 0 okay)

  Test Summary Report
  -------------------
  test-skip (Wstat: 0 Tests: 2 Failed: 1)
    Failed test:  1
  Files=1, Tests=2,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

But node-tap has a nicer warning:

  $ tap ./test-skip
  ./test-skip ........................................... 0/2
    not ok failing

    Skipped: 1
       : skipping

  total ................................................. 0/2

    0 passing (39.088ms)
    1 pending
    1 failing

Similarly, node-tap does a better job handling YAML blocks [1]:

  $ ./test-yaml
  TAP version 13
  1..4
  ok 1 - success diagnostic
  # success
  not ok 2 - failure diagnostic
  # failure
  ok 3 - success YAML
    ---
    message: success
    ...
  not ok 4 - failure YAML
    ---
    message: failure
    ...

Prove either shows no diagnostics:

  $ prove test-yaml
  test-yaml .. Failed 2/4 subtests

  Test Summary Report
  -------------------
  test-yaml (Wstat: 0 Tests: 4 Failed: 2)
    Failed tests:  2, 4
  Files=1, Tests=4,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

or it shows all the diagnostics (even for successful tests):

  $ prove --comments test-yaml
  test-yaml .. 1/?
  # success
  # failure
  test-yaml .. Failed 2/4 subtests

  Test Summary Report
  -------------------
  test-yaml (Wstat: 0 Tests: 4 Failed: 2)
    Failed tests:  2, 4
  Files=1, Tests=4,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

I can't find a way to get Prove to show the YAML blocks.  node-tap, on
the other hand, does the right thing with YAML blocks:

  $ tap ./test-yaml
  ./test-yaml ........................................... 2/4
    not ok failure diagnostic
    not ok failure YAML
      message: failure

  total ................................................. 2/4

    2 passing (42.409ms)
    2 failing

We don't use YAML blocks at the moment, but I'm going to transition to
them later.

[1]: http://testanything.org/tap-version-13-specification.html#yaml-blocks

Signed-off-by: W. Trevor King <wking@tremily.us>

Makefile

@@ -1,5 +1,6 @@
 PREFIX ?= $(DESTDIR)/usr
 BINDIR ?= $(DESTDIR)/usr/bin
+TAP ?= tap
 
 BUILDTAGS=
 RUNTIME ?= runc
@@ -39,7 +40,7 @@ clean:
 	rm -f oci-runtime-tool runtimetest *.1 $(VALIDATION_TESTS)
 
 localvalidation:
-	RUNTIME=$(RUNTIME) prove $(VALIDATION_TESTS)
+	RUNTIME=$(RUNTIME) $(TAP) $(VALIDATION_TESTS)
 
 .PHONY: validation-executables
 validation-executables: $(VALIDATION_TESTS)

README.md

@@ -30,72 +30,84 @@ INFO[0000] Bundle validation succeeded.
 
 ## Testing OCI runtimes
 
-The runtime validation suite uses [`prove`][prove], which is packaged for most distributions (for example, it is in [Debian's `perl` package][debian-perl] and [Gentoo's `dev-lang/perl` package][gentoo-perl]).
-If you cannot install `prove`, you can probably run the test suite with another [TAP consumer][tap-consumers], although you'll have to edit the [`Makefile`](Makefile) to replace `prove`.
+The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
+If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:
+
+```console
+$ npm install tap
+```
 
 ```console
 $ make runtimetest validation-executables
-$ sudo make RUNTIME=runc localvalidation
-RUNTIME=runc prove validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
-validation/linux_rootfs_propagation_shared.t ...... Failed 1/19 subtests
-validation/create.t ............................... ok
-validation/default.t .............................. ok
-validation/linux_readonly_paths.t ................. ok
-validation/linux_masked_paths.t ................... Failed 1/19 subtests
-validation/mounts.t ............................... ok
-validation/process.t .............................. ok
-validation/root_readonly_false.t .................. ok
-validation/linux_sysctl.t ......................... ok
-validation/linux_devices.t ........................ ok
-validation/linux_gid_mappings.t ................... Failed 1/19 subtests
-validation/process_oom_score_adj.t ................ ok
-validation/process_capabilities.t ................. ok
-validation/process_rlimits.t ...................... ok
-validation/root_readonly_true.t ................... ok
-validation/linux_rootfs_propagation_unbindable.t .. failed to create the container
+RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
+validation/linux_rootfs_propagation_shared.t ........ 18/19
+  not ok rootfs propagation
+
+validation/create.t ................................... 4/4
+validation/default.t ................................ 19/19
+validation/linux_readonly_paths.t ................... 19/19
+validation/linux_masked_paths.t ..................... 18/19
+  not ok masked paths
+
+validation/mounts.t ................................... 0/1
+  Skipped: 1
+     TODO: mounts generation options have not been implemented
+
+validation/process.t ................................ 19/19
+validation/root_readonly_false.t .................... 19/19
+validation/linux_sysctl.t ........................... 19/19
+validation/linux_devices.t .......................... 19/19
+validation/linux_gid_mappings.t ..................... 18/19
+  not ok gid mappings
+
+validation/process_oom_score_adj.t .................. 19/19
+validation/process_capabilities.t ................... 19/19
+validation/process_rlimits.t ........................ 19/19
+validation/root_readonly_true.t ...................failed to create the container
 rootfsPropagation=unbindable is not supported
 exit status 1
-validation/linux_rootfs_propagation_unbindable.t .. Dubious, test returned 1 (wstat 256, 0x100)
-No subtests run
-validation/hostname.t ............................. ok
-validation/linux_uid_mappings.t ................... failed to create the container
+validation/root_readonly_true.t ..................... 19/19
+validation/linux_rootfs_propagation_unbindable.t ...... 0/1
+  not ok validation/linux_rootfs_propagation_unbindable.t
+    timeout: 30000
+    file: validation/linux_rootfs_propagation_unbindable.t
+    command: validation/linux_rootfs_propagation_unbindable.t
+    args: []
+    stdio:
+      - 0
+      - pipe
+      - 2
+    cwd: /…/go/src/github.com/opencontainers/runtime-tools
+    exitCode: 1
+
+validation/hostname.t ...................failed to create the container
 User namespace mappings specified, but USER namespace isn't enabled in the config
 exit status 1
-validation/linux_uid_mappings.t ................... Dubious, test returned 1 (wstat 256, 0x100)
-No subtests run
-
-Test Summary Report
--------------------
-validation/linux_rootfs_propagation_shared.t    (Wstat: 0 Tests: 19 Failed: 1)
-  Failed test:  16
-validation/linux_masked_paths.t                 (Wstat: 0 Tests: 19 Failed: 1)
-  Failed test:  13
-validation/linux_gid_mappings.t                 (Wstat: 0 Tests: 19 Failed: 1)
-  Failed test:  19
-validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
-  Non-zero exit status: 1
-  Parse errors: No plan found in TAP output
-validation/linux_uid_mappings.t                 (Wstat: 256 Tests: 0 Failed: 0)
-  Non-zero exit status: 1
-  Parse errors: No plan found in TAP output
-Files=18, Tests=271, 31 wallclock secs ( 0.06 usr  0.01 sys +  0.52 cusr  0.21 csys =  0.80 CPU)
-Result: FAIL
-make: *** [Makefile:42: localvalidation] Error 1
+validation/hostname.t ............................... 19/19
+validation/linux_uid_mappings.t ....................... 0/1
+  not ok validation/linux_uid_mappings.t
+    timeout: 30000
+    file: validation/linux_uid_mappings.t
+    command: validation/linux_uid_mappings.t
+    args: []
+    stdio:
+      - 0
+      - pipe
+      - 2
+    cwd: /…/go/src/github.com/opencontainers/runtime-tools
+    exitCode: 1
+
+total ............................................. 267/273
+
+
+  267 passing (31s)
+  1 pending
+  5 failing
+
+make: *** [Makefile:43: localvalidation] Error 1
 ```
 
-If you are confident that the `validation/*.t` are current, you can run `prove` (or your preferred TAP consumer) directly to avoid unnecessary rebuilds:
-
-```console
-$ sudo RUNTIME=runc prove -Q validation/*.t
-
-Test Summary Report
--------------------
-…
-Files=18, Tests=271, 31 wallclock secs ( 0.07 usr  0.01 sys +  0.51 cusr  0.22 csys =  0.81 CPU)
-Result: FAIL
-```
-
-And you can run an individual test executable directly:
+You can also run an individual test executable directly:
 
 ```console
 $ RUNTIME=runc validation/default.t
@@ -122,10 +134,45 @@ ok 19 - gid mappings
 1..19
 ```
 
+If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
+For example, with [`prove`][prove]:
+
+```console
+$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation
+RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
+failed to create the container
+rootfsPropagation=unbindable is not supported
+exit status 1
+failed to create the container
+User namespace mappings specified, but USER namespace isn't enabled in the config
+exit status 1
+
+Test Summary Report
+-------------------
+validation/linux_rootfs_propagation_shared.t    (Wstat: 0 Tests: 19 Failed: 1)
+  Failed test:  16
+validation/linux_masked_paths.t                 (Wstat: 0 Tests: 19 Failed: 1)
+  Failed test:  13
+validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
+  Non-zero exit status: 1
+  Parse errors: No plan found in TAP output
+validation/linux_uid_mappings.t                 (Wstat: 256 Tests: 0 Failed: 0)
+  Non-zero exit status: 1
+  Parse errors: No plan found in TAP output
+validation/linux_gid_mappings.t                 (Wstat: 0 Tests: 19 Failed: 1)
+  Failed test:  19
+Files=18, Tests=271,  6 wallclock secs ( 0.06 usr  0.01 sys +  0.59 cusr  0.24 csys =  0.90 CPU)
+Result: FAIL
+make: *** [Makefile:43: localvalidation] Error 1
+```
+
 [bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
 [config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
-[debian-perl]: https://packages.debian.org/stretch/perl
-[gentoo-perl]: https://packages.gentoo.org/packages/dev-lang/perl
+[debian-node-tap]: https://packages.debian.org/stretch/node-tap
+[debian-nodejs]: https://packages.debian.org/stretch/nodejs
+[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
+[node-tap]: http://www.node-tap.org/
+[npm]: https://www.npmjs.com/
 [prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
 [runC]: https://github.com/opencontainers/runc
 [runtime-spec]: https://github.com/opencontainers/runtime-spec