Stress-testing caused to stuck on OS X

[josser]

• Version: v6.5.0
• Platform: Darwin josser-mbp 15.6.0 Darwin Kernel Version 15.6.0: Mon Aug 29 20:21:34 PDT 2016; root:xnu-3248.60.11~1/RELEASE_X86_64 x86_64
• Subsystem: http

Trying to do some stress-tests using ab (yes, I know it's a bit primitive idea) but it looks like there is some issue in OSX-specific code.

So, here is example code:

const cluster = require('cluster');
const os = require('os');
const http = require('http');

if (cluster.isMaster) {
  const cpuCount = os.cpus().length;

  for (let i = 0; i < cpuCount; i++) {
    cluster.fork();
  }
} else {  
  var server = http.createServer( (req, res) => {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('okay');
  });

  server.listen(3000);
}

 ab -n 10000 -c 8 http://localhost:3000/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 6386 requests completed

The most interesting thing is that this is happening not always but usually on second ab run.
So, if you don't get error on first run, just try again as quickly as you can.
Issue is not reproducible on linux.

In console.app I see only one message occurring which correlates with ab in case of error:
Here it is:

07.09.16 17:07:07,000 kernel[0]: ALF: ifnet_get_address_list_family error 12

Maybe this can help for investigation.

Thank you and sorry for my bad english.

[evanlucas]

Hi @josser. Thanks for opening the issue. Can you run ulimit -a and paste the output here? Thanks!

[josser]

Here it is:

ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                256

[evanlucas]

Yea, OS X has a pretty low default amount for file descriptors. Have you tried bumping that up? (Something like ulimit -n 4000 and then trying again?

[josser]

@evanlucas Yes, i have tried to adjust it to 10000 using this command:

ulimit -n 10000

I'm not sure where is correct place to set it. So I set it in both terminals (with node and with ab), because it looks like this setting applies per terminal window. (or shell session).
But this not help, it stuck with same conditions.

[evanlucas]

@josser just curious, after you set it, have you confirmed that it is set (ulimit -n)? I know there are some times where you have to use launchctl to set it before you can use ulimit to set it.

[josser]

@evanlucas Yes, I think I can confirm it set. When I run ulimit -a again I see new value:

ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                10000

[mscdex]

I thought there was always some issue with ab on OS X? Did you try using wrk or similar instead?

[josser]

@mscdex Cool, looks like there is something wrong with ab on mac.

wrk -t 8 -d 20 http://localhost:3000
Running 20s test @ http://localhost:3000
  8 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   181.67us   27.05us   2.31ms   87.50%
    Req/Sec     5.43k   168.34     6.01k    75.87%
  868059 requests in 20.10s, 122.52MB read
Requests/sec:  43186.89
Transfer/sec:      6.10MB

Looks good, thank you!