Sentinel connection error not emitting

[madeinjam]

Hi!

I'm unable to get an error when ioredis can not connect to sentinel. Code:

var Redis  = require('ioredis');
var redis = new Redis({
  sentinels             : [{ host:'127.0.0.2', port: 26379 }],
  name                  : 'mymaster',
  connectTimeout : 1000,
  db                       : 0,
  sentinelRetryStrategy : (times) => {
    if (times >= 3) {
      return null;
    }
    var delay = Math.min(((times-1) * 10) * (times * 10), 2000);
    return delay;
  }
});

redis.on('error', (error) => {
  console.log('Error opening connection');
});

redis.once('connect', () => {
  console.log('Connection established');
});

redis.once('ready', () => {
  console.log('Connection ready');
  redis.disconnect();
});

redis.once('close', () => {
  console.log('Connection closed');
});

redis.once('end', () => {
  console.log('Connection ended');
});

redis.once('reconnecting', () => {
  console.log('Reconnecting');
});

Output:

DEBUG=ioredis:* node sentinel-test.js
  ioredis:redis status[localhost:6379]: [empty] -> connecting +0ms
  ioredis:redis status[127.0.0.2:26379]: [empty] -> connecting +3ms
  ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster) +3ms
  ioredis:redis status[127.0.0.2:26379]: connecting -> close +1s
  ioredis:connection skip reconnecting because `retryStrategy` is not a function +1ms
  ioredis:redis status[127.0.0.2:26379]: close -> end +0ms
  ioredis:SentinelConnector failed to connect to sentinel 127.0.0.2:26379 because Error: Connection is closed. +4ms
  ioredis:SentinelConnector All sentinels are unreachable. Retrying from scratch after 0 +1ms
  ioredis:redis status[127.0.0.2:26379]: [empty] -> connecting +1ms
  ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster) +1ms
  ioredis:redis status[127.0.0.2:26379]: connecting -> close +1s
  ioredis:connection skip reconnecting because `retryStrategy` is not a function +0ms
  ioredis:redis status[127.0.0.2:26379]: close -> end +0ms
  ioredis:SentinelConnector failed to connect to sentinel 127.0.0.2:26379 because Error: Connection is closed. +1ms
  ioredis:SentinelConnector All sentinels are unreachable. Retrying from scratch after 200 +0ms
  ioredis:redis status[127.0.0.2:26379]: [empty] -> connecting +202ms
  ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster) +0ms
  ioredis:redis status[127.0.0.2:26379]: connecting -> close +1s
  ioredis:connection skip reconnecting because `retryStrategy` is not a function +1ms
  ioredis:redis status[127.0.0.2:26379]: close -> end +0ms
  ioredis:SentinelConnector failed to connect to sentinel 127.0.0.2:26379 because Error: Connection is closed. +0ms
  ioredis:SentinelConnector All sentinels are unreachable and retry is disabled, emitting error... +0ms

The output is only ioredis debug, no consoles log, the error event was never emitted, is these a bug or am i doing something wrong?

Thanks, awesome work!

[luin]

Yes, it's a bug. Released in 1.11.1, now it will emit error event when all sentinels are unreachable. Thank you for pointing it out! 🚀

[madeinjam]

Awesome, thanks!

[Gorokhov]

this seems to work properly only in the case if sentinelRetryStrategy is given as above.
With basic configuration ex:

client = new Redis({
                sentinels: config.sentinels,
                name: config.sentinelName,
                connectTimeout: 1000
            });

the error is never fired, maybe this is expected but not intuative.

[luin]

@Gorokhov Yes, by default ioredis will keep reconnecting to Redis until the connection has been made. This behaviour is important when it's used in production, that even redis servers are down for several seconds, ioredis will still connect to them once they are online again.

[kunth]

Hi luin, I have the same question, I already set my ioredis version to 1.14.0 or 1.11.1, but neither does it work out.
My sentinel(redis) version is 2.8.19, the code as follows:

var Redis = require('ioredis');

var opts = {
name: 'rnode_53',
sentinels: [
//{host: 'any.org', port: 9600},
//{host: 'any.org', port: 9601},
{host: 'any.org', port: 9602}
],
//retryStrategy: function(times) {
// var delay = 2000;
// var delay = Math.min(times * 2, 2000);
// return delay;
//},
sentinelRetryStrategy : function(times) {
//var delay = Math.min(times * 2, 1000);
var delay = 1000;
return delay;
}
}
var redis = new Redis(opts);
redis.on('error', function(err){
console.log('catch a error: ' + err);
});

I can get error from retryStrategy(if sentinel runs well and related nodes down), but not sentinelRetryStrategy(if sentinel down).
Any suggestions?

[luin]

@kunth sentinelRetryStrategy is only invoked when all the sentinel servers are not reachable.

[kunth]

yeah, I noticed that. I killed all sentinels when I try this.

[luin]

@kunth and sentinelRetryStrategy is still not invoked? Could you please enable the debug mode (DEBUG=ioredis:* node yourapp.js) and post the logs here?

[kunth]

@luin , sentinelRetryStrategy never invoked in my case.
I had erased sensitive info(shown as any.org) and my own log info
the debug info:

ioredis:redis status[localhost:6379]: [empty] -> connecting +0ms
ioredis:redis status[any.org:9602]: [empty] -> connecting +2ms
ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,rnode_53) +2ms
ioredis:connection error: Error: connect ECONNREFUSED +1s
ioredis:redis status[any.org:9602]: connecting -> close +1ms
ioredis:connection skip reconnecting because retryStrategy is not a function +0ms
ioredis:redis status[any.org:9602]: close -> end +0ms
ioredis:SentinelConnector failed to connect to sentinel any.org:9602 because Error: Connection is closed. +1ms
ioredis:SentinelConnector All sentinels are unreachable. Retrying from scratch after 1000 +0ms
ioredis:redis status[any.org:9602]: [empty] -> connecting +1s
ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,rnode_53) +1ms
ioredis:connection error: Error: connect ECONNREFUSED +1s
ioredis:redis status[any.org:9602]: connecting -> close +0ms
ioredis:connection skip reconnecting because retryStrategy is not a function +0ms
ioredis:redis status[any.org:9602]: close -> end +0ms
ioredis:SentinelConnector failed to connect to sentinel any.org:9602 because Error: Connection is closed. +0ms
ioredis:SentinelConnector All sentinels are unreachable. Retrying from scratch after 1000 +0ms
ioredis:redis status[any.org:9602]: [empty] -> connecting +1s
ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,rnode_53) +0ms

[luin]

That helps. It seems sentinelRetryStrategy is invoked since in your case sentinelRetryStrategy always returns 1000 and the logs says Retrying from scratch after 1000. Have you tried adding a console.log to the sentinelRetryStrategy function?

[kunth]

@luin , yeah I manually set the delay to 1000 in sentinelRetryStrategy function.
And I can see the log in sentinelRetryStrategy function, but why the "error event" never triggered?

[luin]

ioredis only emits error event when sentinelRetryStrategy returns a non-number value. end and error event means the connection is failed and you should manually restart a new connection.

[kunth]

Well, so there is a difference between retryStrategy and sentinelRetryStrategy.
As to retryStrategy, no matter what value is set to delay, the redis.on('error') wolud be triggered. However in sentinelRetryStrategy, only set delay to non-number value would emit error .
Is that the way?

[luin]

Yes, that's correct.

[kunth]

Well, a little weird. Thanks very much, @luin , you save me lots of time.