System.TimeoutException: The operation has timed out.

[chsakell]

Hello,

We are getting System.TimeoutException: The operation has timed out. exceptions when using QueueBind or UnBindQueue methods. The issue happens only in high traffic hours in an ASP.NET Core SignalR 3.1 application hosted in linux containers - OpenShift.

Exception:

System.TimeoutException: The operation has timed out.
   at RabbitMQ.Util.BlockingCell`1.WaitForValue(TimeSpan timeout)
   at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)
   at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)
   at RabbitMQ.Client.Framing.Impl.Model.QueueUnbind(String queue, String exchange, String routingKey, IDictionary`2 arguments)
   at RabbitMQ.Client.Impl.AutorecoveringModel.QueueUnbind(String queue, String exchange, String routingKey, IDictionary`2 arguments)
   at Customer.PushService.Infrastructure.RabbitMq.Bindings.RabbitMqCustomerBindingsService.UnBindQueue(String routingKey) 

When a user subscribes to a Signalr method, a binding is created in RabbitMQ.

private void BindQueue(string routingKey)
{
    using var model = _connection.CreateModel();
    model.QueueBind(queueName, exchangeName, routingKey, null);
}

At some point though during high volume, it seems that everything is just stopped, thread pool starvation is created and the above errors are shown.

Searching the code, it seems that it throws the exception in the BlockingCell WaitForValue method.

We don't have any memory leak or cpu issue on the ASP.NET Core app, so can you tell a reason why this happens?
Of course there's a lock statement in the ModelBase, and the GetReply is the one that throws the exception, but what does this actually mean? Is it possible that multiple concurrent requests to QueueBind or QueueUnBind cause this issue?

Thanks

[sungam3r]

I think this problem has been already discussed before. See #860 and related issues with ThreadPool.SetMinThreads.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Or the node is genuinely overloaded and some client operations time out. Server node infrastructure and thread activity metrics can confirm this.

Finally, reducing the churn in your topology will result in drastically fewer binding or unbinding operations to begin with.

[chsakell]

Ok @michaelklishin we will check the metrics for sure. We had also tried the min threads trick but didn't get us far.

Can you tell if one of the followings could help somehow?

• create a pool of connections from which the clients create individual channels instead of using a single connection
• use QueueBindNoWait instead of QeueuBind. The no wait option seems that doesn't lock/wait so I guess we would see a better behavior. Keep in mind that the worst part is the thread pool starvation that's been created during the issue and dramatically affects app's performance.

[lukebakken]

@chsakell version 7 of this client should help with this scenario. If you have time to test pre-releases, we would appreciate it:

https://www.nuget.org/packages/RabbitMQ.Client/7.0.0-alpha.6