RabbitMQ consumer cannot connect back to cluster after one node down

[Pramod34]

Hi Team,

We have a production environment with 3 cluster nodes and everything is working fine when all the 3 nodes in RabbitMQ cluster. When one RabbitMQ node is down, consumer is unable to change to other 2 working RabbitMQ nodes. We have provided all 3 nodes IPAddress in our consumer and also we have set AutomaticRecoveryEnabled, NetworkRecoveryInterval to true.

When everything is working normal, consumer count is shown in the admin panel for that consumer queue, but once restarted queue is still appears but consumer count is shown as zero.

Can you please let us know any possible issue or approach to flow?

Thanks,
Pramod

[lukebakken]

When one RabbitMQ node is down

Questions:

• What version of RabbitMQ and Erlang?
• What version of the .NET library?
• What version of .NET?
• What operating system and version is being used to run RabbitMQ and your client application?
• How are you bringing a node down?

[Pramod34]

Hi,

We are using RabbitMQ version 3.13.0 and Erlang version 26.2.3.
.NET Library - 5.2.0
.NET Version - 4.5.1
Operating system we are running RabbitMQ on Windows Server 2019 with 8GB RAM. We are running application on different server like Windows 10, 11 and windows server 2012, 2016

We brought down server by stopping RabbitMQ service from services.msc in windows machine.

We don't have a docker environment. We are using VM's with windows server installed on them.

Please help me out on this.

Thanks,
Pramod

[lukebakken]

Unfortunately we do not support version 5.x of this library any more. .NET 4.5 is no longer supported, and unless you have an extended support contract with Microsoft, neither is Windows Server 2019 it seems (https://learn.microsoft.com/en-us/lifecycle/products/windows-server-2019).

I strongly suggest using a load balancer.

[lukebakken]

@Pramod34 please see this complete docker compose project:

https://github.com/lukebakken/rabbitmq-dotnet-client-1584

Running make up will bring up a RabbitMQ cluster, with a .NET consuming and producing application.

I have tested "force close" via the Management UI, as well as stopping a node to which an application is connected, via docker compose stop rmqX.

The applications do recover, however, I did have to subscribe to the ConnectionShutdown, ModelShutdown and RecoverySucceeded events to get things restored "just right". Without waiting long enough for connection.IsOpen to return true, sometimes recovery just didn't work right. I'm not exactly sure why at this time.

I am going to create a branch in this project to test reconnection behavior using version 7.0.0-rc.1 of this client library to see what the behavior is like.

[lukebakken]

https://github.com/lukebakken/rabbitmq-dotnet-client-1584/tree/rabbitmq-dotnet-client-v7

[lukebakken]

@Pramod34 - please note that connection recovery is more reliable when applications are configured to use HAproxy. See this branch: https://github.com/lukebakken/rabbitmq-dotnet-client-1584/blob/haproxy/producer/Program.cs#L10-L11