Temporarily disabling node-test-commit-arm-fanned

[rvagg]

Too many Pis are offline and the jobs are taking too long.

• 2 Pi1's online: https://ci.nodejs.org/label/pi1-docker/
• 5 Pi2's online: https://ci.nodejs.org/label/pi2-docker/
• 2 Pi3's online: https://ci.nodejs.org/label/pi3-docker/

Lots of perma-red disables in there but a bunch of other disconnects for other reasons. I can ping a little over 1/2 of them, so they're in a poor state and need some love. One of the nfs export disks is full which explains a lot of them being offline.

[mhdawson]

Off topic but @rvagg I'm wondering if you have a feel if PI4's are fast enough that we could avoid "fanned" and the need for so many machines?

[rvagg]

@mhdawson not entirely sure but I think until Pi gets alternative storage it's always going to be a problem for workloads like ours. SD is just not a fast medium, but it's cheap, which is why they do it.

I think I got a 4 to play with but never ran a Node build on it, maybe I should try.

[mhdawson]

@rvagg is that true even with the nfs mounts you've been using?

[rvagg]

@mhdawson nfs seems to speed things up a little but it's not a silver bullet since we have a fleet of machines all hammering the same network and same disk with small file reads and writes, and nfs isn't exactly an efficient protocol either.

maybe the cpu and ram improvements in a pi4 would improve the situation though.

[mhdawson]

@rvagg thanks, understood that if we still have large numbers of machines its still a problem. I don't have a PI 4 yet to experiment but I'll have to think about getting one.

[rvagg]

Tended to the Pi cluster today and arm-fanned is now re-enabled with all machines running, connected, updated, cleaned and some runs for major release lines pushed through to re-prime caches and look for hiccups.

Noteworthy items:

• I've taken half of each type of Pi offline, they are marked as "RESERVE" with a note that they should be enabled when others are disabled. I'm doing this to save disk on the shared NFS host. One we move past Node 10 we'll ditch the Pi 1's and have lots more disk to play with. @Trott you're the most active with taking Pi's offline - feel free to enable a RESERVE Pi when you do to add more bandwidth. You can see the list easily by label: pi1-docker, pi2-docker & pi3-docker.
• There was a bug in the arm-fanned config that was causing test-binary-arm and test-binary-arm-12+ to be run for Node 10, so Pi 2 & 3 tests were running twice. That should be fixed now.
• addons/openssl-binding/test very flaky on ARM node#30786 is outstanding and will probably flare up again now ARMv7 is back in CI. There's some suggestions in there about how it might be addressed, additional perspectives or solutions (in PR form!) welcome.
• We seem to have introduced a perma-fail test in the time that this job was offline known_issues/test-crypto-authenticated-stream times out on Pi 2 and 3 from master (Pi 1 not run for master). See below. Ping to @nodejs/crypto @nodejs/testing on this I guess.

19:52:57 not ok 655 known_issues/test-crypto-authenticated-stream
19:52:57   ---
19:52:57   duration_ms: 240.118
19:52:57   severity: fail
19:52:57   exitcode: -15
19:52:57   stack: |-
19:52:57     timeout
19:52:57   ...

[bnoordhuis]

known_issues/test-crypto-authenticated-stream times out on Pi 2 and 3

Not really a surprise. I opened nodejs/node#32636 to disable it on ARM.