benchmark: make compare.R easier to understand

[AndreasMadsen]

As talked about in #18112 (comment) this shows more clearly the variance of each comparison. This should also help us prevent over-running the benchmarks. If you see an accuracy of ±0.1% then properly you could spend fewer iterations running that ;)

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• commit message follows commit guidelines

Affected core subsystem(s)

benchmark

example output:

                                                                                  confidence improvement accuracy (*)    (**)   (***)
 fs/bench-readdir.js n=10000                                                                     -7.20 %       ±9.94% ±13.23% ±17.22%
 fs/bench-readdirSync.js n=10000                                                                 -0.46 %       ±6.97%  ±9.27% ±12.08%
 fs/bench-realpath.js pathType="relative" n=10000                                                 1.62 %       ±4.25%  ±5.65%  ±7.35%
 fs/bench-realpath.js pathType="resolved" n=10000                                                -1.68 %       ±3.89%  ±5.17%  ±6.74%
 fs/bench-realpathSync.js pathType="relative" n=10000                                             0.72 %       ±5.87%  ±7.81% ±10.18%
 fs/bench-realpathSync.js pathType="resolved" n=10000                                             0.16 %       ±1.61%  ±2.15%  ±2.80%
 fs/bench-stat.js statType="fstat" n=200000                                                      -2.39 %       ±4.50%  ±6.00%  ±7.83%
 fs/bench-stat.js statType="lstat" n=200000                                                      -2.94 %       ±5.34%  ±7.11%  ±9.25%
 fs/bench-stat.js statType="stat" n=200000                                                        2.30 %       ±4.29%  ±5.72%  ±7.45%
 fs/bench-statSync.js statSyncType="fstatSync" n=1000000                                         -1.15 %       ±5.69%  ±7.57%  ±9.85%
 fs/bench-statSync.js statSyncType="lstatSync" n=1000000                                          0.47 %       ±2.15%  ±2.90%  ±3.84%
 fs/bench-statSync.js statSyncType="statSync" n=1000000                                           0.62 %       ±2.99%  ±3.98%  ±5.19%
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType="asc"                    1.08 %       ±2.96%  ±3.94%  ±5.13%
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType="buf"                   -1.86 %       ±3.85%  ±5.13%  ±6.68%
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType="utf"                   -1.09 %       ±2.29%  ±3.04%  ±3.96%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType="asc"                -1.70 %       ±4.55%  ±6.07%  ±7.95%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType="buf"                -0.51 %       ±3.54%  ±4.71%  ±6.14%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType="utf"                 0.21 %      ±10.19% ±13.56% ±17.65%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType="asc"                   -1.84 %       ±3.14%  ±4.17%  ±5.43%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType="buf"                   -2.14 %       ±4.09%  ±5.44%  ±7.08%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType="utf"                    1.17 %       ±3.53%  ±4.70%  ±6.11%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType="asc"                   1.25 %       ±7.25%  ±9.65% ±12.56%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType="buf"            *      6.19 %       ±5.13%  ±6.86%  ±9.01%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType="utf"                   2.12 %       ±6.49%  ±8.64% ±11.25%
 fs/readfile.js concurrent=1 len=1024 dur=5                                                      -0.33 %       ±3.64%  ±4.84%  ±6.31%
 fs/readfile.js concurrent=1 len=16777216 dur=5                                                  -3.03 %       ±5.35%  ±7.12%  ±9.27%
 fs/readfile.js concurrent=10 len=1024 dur=5                                                     -0.98 %       ±2.13%  ±2.83%  ±3.68%
 fs/readfile.js concurrent=10 len=16777216 dur=5                                                  0.14 %       ±1.72%  ±2.29%  ±2.98%
 fs/readFileSync.js n=600000                                                                     -4.79 %       ±7.59% ±10.10% ±13.17%
 fs/write-stream-throughput.js size=1024 encodingType="asc" dur=5                                -1.60 %       ±8.77% ±11.66% ±15.18%
 fs/write-stream-throughput.js size=1024 encodingType="buf" dur=5                                 0.39 %      ±12.28% ±16.34% ±21.27%
 fs/write-stream-throughput.js size=1024 encodingType="utf" dur=5                                -0.77 %       ±5.77%  ±7.69% ±10.03%
 fs/write-stream-throughput.js size=1048576 encodingType="asc" dur=5                              0.18 %       ±0.99%  ±1.31%  ±1.71%
 fs/write-stream-throughput.js size=1048576 encodingType="buf" dur=5                       *     23.42 %      ±23.37% ±31.12% ±40.58%
 fs/write-stream-throughput.js size=1048576 encodingType="utf" dur=5                              2.73 %       ±7.07%  ±9.45% ±12.41%
 fs/write-stream-throughput.js size=2 encodingType="asc" dur=5                                    0.74 %      ±11.24% ±14.96% ±19.47%
 fs/write-stream-throughput.js size=2 encodingType="buf" dur=5                                   -2.84 %       ±5.64%  ±7.52%  ±9.81%
 fs/write-stream-throughput.js size=2 encodingType="utf" dur=5                             *    -15.35 %      ±14.18% ±18.87% ±24.58%
 fs/write-stream-throughput.js size=65535 encodingType="asc" dur=5                               -3.07 %      ±10.67% ±14.21% ±18.53%
 fs/write-stream-throughput.js size=65535 encodingType="buf" dur=5                         *    -12.87 %      ±11.94% ±15.88% ±20.67%
 fs/write-stream-throughput.js size=65535 encodingType="utf" dur=5                                3.28 %       ±9.81% ±13.08% ±17.06%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case there are 41 comparisons, you can thus
expect the following amount of false-positive results:
  2.05 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.41 false positives, when considering a   1% risk acceptance (**, ***),
  0.04 false positives, when considering a 0.1% risk acceptance (***)

[AndreasMadsen]

/cc @joyeecheung
/cc @jasnell @Matteo - who have been my "I don't understand statistic" test subjects.

[joyeecheung]

Not a R expert, but LGTM if benchmark CI is happy.

[joyeecheung]

Benchmark CI: https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/103/

Looking at https://github.com/nodejs/benchmarking/blob/master/experimental/benchmarks/community-benchmark/run.sh I think this should be using the new R script for the results, cc @gareth-ellis

[joyeecheung]

                                               confidence improvement accuracy (*)    (**)   (***)
 arrays/var-int.js n=25 type="Array"                           4.97 %      ±13.05% ±17.96% ±24.65%
 arrays/var-int.js n=25 type="Buffer"                         -0.85 %      ±12.22% ±16.76% ±22.88%
 arrays/var-int.js n=25 type="Float32Array"                   -0.31 %       ±0.43%  ±0.58%  ±0.80%
 arrays/var-int.js n=25 type="Float64Array"                    0.34 %       ±0.50%  ±0.69%  ±0.95%
 arrays/var-int.js n=25 type="Int16Array"                      3.92 %       ±9.76% ±14.00% ±20.56%
 arrays/var-int.js n=25 type="Int32Array"                     -0.21 %       ±0.48%  ±0.66%  ±0.93%
 arrays/var-int.js n=25 type="Int8Array"                      -5.21 %       ±9.54% ±13.70% ±20.15%
 arrays/var-int.js n=25 type="Uint16Array"                    -0.14 %      ±12.71% ±17.42% ±23.73%
 arrays/var-int.js n=25 type="Uint32Array"                    -0.79 %       ±1.49%  ±2.13%  ±3.09%
 arrays/var-int.js n=25 type="Uint8Array"                      4.23 %      ±10.50% ±15.08% ±22.17%
 arrays/zero-float.js n=25 type="Array"                       -1.47 %       ±4.95%  ±7.10% ±10.41%
 arrays/zero-float.js n=25 type="Buffer"                       3.84 %       ±9.00% ±12.92% ±18.98%
 arrays/zero-float.js n=25 type="Float32Array"                 0.30 %       ±0.53%  ±0.73%  ±0.99%
 arrays/zero-float.js n=25 type="Float64Array"                -0.07 %       ±1.77%  ±2.50%  ±3.59%
 arrays/zero-float.js n=25 type="Int16Array"                   2.44 %       ±7.92% ±11.26% ±16.31%
 arrays/zero-float.js n=25 type="Int32Array"                  -0.04 %       ±0.63%  ±0.88%  ±1.24%
 arrays/zero-float.js n=25 type="Int8Array"                    0.10 %       ±2.76%  ±3.78%  ±5.16%
 arrays/zero-float.js n=25 type="Uint16Array"                 -0.72 %       ±1.92%  ±2.74%  ±3.98%
 arrays/zero-float.js n=25 type="Uint32Array"                  0.67 %       ±0.92%  ±1.27%  ±1.76%
 arrays/zero-float.js n=25 type="Uint8Array"                  -7.27 %      ±11.99% ±17.15% ±25.07%
 arrays/zero-int.js n=25 type="Array"                          2.16 %       ±8.68% ±11.95% ±16.41%
 arrays/zero-int.js n=25 type="Buffer"                         3.86 %       ±8.99% ±12.91% ±18.97%
 arrays/zero-int.js n=25 type="Float32Array"                  -0.26 %       ±0.57%  ±0.78%  ±1.07%
 arrays/zero-int.js n=25 type="Float64Array"                  -0.42 %       ±0.65%  ±0.90%  ±1.25%
 arrays/zero-int.js n=25 type="Int16Array"                     4.87 %       ±9.30% ±13.36% ±19.64%
 arrays/zero-int.js n=25 type="Int32Array"                     0.16 %       ±1.78%  ±2.51%  ±3.61%
 arrays/zero-int.js n=25 type="Int8Array"                      4.88 %       ±9.88% ±14.18% ±20.82%
 arrays/zero-int.js n=25 type="Uint16Array"                    3.03 %       ±9.31% ±13.27% ±19.29%
 arrays/zero-int.js n=25 type="Uint32Array"                    1.28 %       ±2.26%  ±3.24%  ±4.74%
 arrays/zero-int.js n=25 type="Uint8Array"                     4.17 %       ±9.60% ±13.79% ±20.27%

Be aware that when doing many comparisions the risk of a false-positive
result increases. In this case there are 30 comparisions, you can thus
expect the following amount of false-positive results:
  1.50 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.30 false positives, when considering a   1% risk acceptance (**, ***),
  0.03 false positives, when considering a 0.1% risk acceptance (***)
Notifying upstream projects of job completion
Finished: SUCCESS

The accuracy there probably means those benchmarks are just not that reliable in nature...

[gareth-ellis]

@joyeecheung / @AndreasMadsen

LGTM, i seem to be having issues with my email notifications at the moment.
I've also noticed in the output that its not particularly easy to confirm what has been done - i should be able to update the script to make it very clear what change we make to the build under test, which should make this a lot clearer.

As there is the extra warning about false positives in the benchmark output, I think we can be sure that the change was in this build - but i think i can make it clearer in the future.!

Note, the job is actually running https://github.com/nodejs/benchmarking/blob/core-benchmark/experimental/benchmarks/community-benchmark/run.sh

I'll get this into master, as that's going to lead to even more confusion (I made some changes to try and reduce output from the build, but I need to also get it to take stderr away, as we have a lot of warnings that make the rest of the output trickier to understand.)

[joyeecheung]

Landed in 368517c, thanks!