Investigate parallel/test-cluster-send-handle-large-payload failure on FreeBSD

[Trott]

• Version: 9.0.0-pre
• Platform: freebsd10-64
• Subsystem: test

New and different failure for test-cluster-send-handle-large-payload, this time on FreeBSD.

https://ci.nodejs.org/job/node-test-commit-freebsd/10969/nodes=freebsd10-64/console

not ok 243 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 1.638
  severity: fail
  stack: |-
    undefined:1
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    
    SyntaxError: Unexpected token a in JSON at position 0
        at JSON.parse (<anonymous>)
        at Pipe.channel.onread (internal/child_process.js:479:28)
  ...

@addaleax

[addaleax]

sigh

Not sure how this can happen, but at least this is an OS I can get access to.

[addaleax]

Stress test to get an idea of just how flaky this is: https://ci.nodejs.org/job/node-stress-single-test/1375/nodes=freebsd10-64/

[BridgeAR]

It happened again recently.

[MylesBorins]

happened again https://ci.nodejs.org/job/node-test-commit-freebsd/15723/nodes=freebsd11-x64/consoleFull

[Trott]

Happened just now: https://ci.nodejs.org/job/node-test-commit-freebsd/16120/nodes=freebsd11-x64/console

[Trott]

Stress test, once in sequence and once with -J for parallelism.

Sequence: https://ci.nodejs.org/job/node-stress-single-test/1787/nodes=freebsd11-x64/
Parallel: https://ci.nodejs.org/job/node-stress-single-test/1788/nodes=freebsd11-x64/

[Trott]

I messed up the parallel stress test. Here it is again, hopefully correctly done this time:

https://ci.nodejs.org/job/node-stress-single-test/1789/nodes=freebsd11-x64/

[Trott]

Parallel results suggest this is an issue about resource constraints and it might be solved (or mitigated) by moving the test to sequential. I'll submit a PR to do that.

ok 1 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 2.853
  ...
ok 2 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 3.294
  ...
ok 3 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 3.316
  ...
ok 4 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 3.172
  ...
not ok 5 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 3.92
  severity: fail
  stack: |-
    undefined:1
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    
    SyntaxError: Unexpected token a in JSON at position 0
        at JSON.parse (<anonymous>)
        at Pipe.channel.onread (internal/child_process.js:490:28)
  ...
ok 6 parallel/test-cluster-send-handle-large-payload
  ---
  duration_ms: 3.236

[Trott]

(Sequential stress test turned up green. It was just the parallel stress test that showed the failures.)

[santigimeno]

I looked into this a while ago and IIRC it seemed to me that it could be related with a bug in the freeBSD kernel where data was silently lost when sending SCM_RIGHTS messages with large payload. See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=181741 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215933.