In which cases the RedisConnectionStateListener.onRedisExceptionCaught method is called?

[Qwex]

We are developing support for password rotation to redis and we need to be able to determine that the connection is unavailable due to incorrect credentials.
We have added a RedisConnectionStateListener to both the connection and the client, but the method RedisConnectionStateListener.onRedisDisconnected that should respond to connection errors is not called if reconnect fails
Should this method be called in this case?
Then how can we determine the exactly reason why the connection is unavailable?

[tishun]

Hey @Qwex ,
could you elaborate more on the scenario you are talking about, what are the steps you take to get to a point where the app attempts to recconect? The more details the better for us to know your use case.

[Qwex]

We are attempting to implement the following scenario (using Scala language):

1. Create a client and set up a connection event listener, which will attempt to intercept connection errors:

object RCSL extends RedisConnectionStateListener {

  override def onRedisExceptionCaught(connection: RedisChannelHandler[_, _], cause: Throwable): Unit = {
    println("connection fail")
    println(cause.getMessage)
  }

}

val redisUri: RedisURI = RedisURI
  .builder(RedisURI.create("some redis uri"))
  .withAuthentication(RCP)

val redisClient = RedisClient.create()
redisClient.addListener(RCSL)

2. Establish a connection using a custom Redis credential provider:

object RCP extends RedisCredentialsProvider {
  override def resolveCredentials(): Mono[RedisCredentials] = {
    val (username, password) = loadCredentials() // some method to get credentials from external resource (in our case Vault)
    Mono.just(new RedisCredentials {
      override def getUsername: String = username

      override def hasUsername: Boolean = true

      override def getPassword: Array[Char] = password.toCharArray

      override def hasPassword: Boolean = true
    })
  }
}

3. Once the connection has been successfully established, we reset the Redis credentials.

4. Then connection is fails. Automatic attempts to recover it begin. We receive logs indicating the reason for the failure: "WRONGPASS: invalid username-password pair". However, we do not receive any messages from RedisConnectionStateListener.onRedisExceptionCaught.

5. Our goal is to identify the cause of the connection issue and, in the case of incorrect credentials, resolve it.

It seems that out original question was not clearly formulated. We need a way to identify the exact cause of the connection loss in order to resolve it. We have not yet found how to do this, but probaly one possible solution is to use the RedisConnectionStateListener. However, this listener does not provide information about the reason for the connection failure.

[tishun]

What steps do you take to execute step 3? Are you using a standalone Redis instance?
If you are running a command could you share it with us?

[Qwex]

We use HashiCorp Vault to install credentials for Redis and generate initial credentials using it. In the step 3, we revoke the credentials through a request to the Vault, after which the connection is terminated. This is a simulation of a situation where credentials suddenly become invalid. To programmatically deal with this, we need to know that the connection was lost due to invalid credentials in order to automatically create new credentials and not create them if the connection was lost for other reasons. We thought that the listener should get the reason for the connection error, but it doesn't.

No, we are connecting to Redis over the network. The application and Redis are running on separate machines.

[tishun]

I am unfortunately not familiar with the HashiCorp Vault and specifically what this flow results in as commands to the Redis server.
To reproduce locally I made a few assumptions:

• the application that uses the Lettuce driver initiates a connection with the server with some valid credentials
• the HashiCorp Vault pushes some new credentials to the server and then executes CLIENTS KILL to make sure no client is left using old (expired) credentials
• at that point the application attempts to reconnect and fails due to invalid credentials

One way to detect such a scenario is to use the EventBus of the Lettuce driver:

        RedisClient redisClient = RedisClient.create(RedisURI.Builder.redis("redis.io", 12000).build());

        EventBus eventBus = redisClient.getResources().eventBus();
        eventBus.get().subscribe(e -> {

            if (e instanceof ReconnectFailedEvent) {
                ReconnectFailedEvent event = (ReconnectFailedEvent) e;
                if (event.getCause().getMessage().contains("WRONGPASS")) {
                    doSomethingAboutIt();
                }
            }
        });

This is not the most elegant solution, but I am unable to come up with a better one right now.

Does that help?