First WebClient request always fails after some idle time #764
Labels
for/user-attention
This issue needs user attention (feedback, rework, etc...)
Comments
martin-tarjanyi
changed the title
|
@martin-tarjanyi I cannot reproduce this. Enable wire log: And in Then reproduce and attach here the log file. |
violetagg
added
the
for/user-attention
|
First request: 21:11 - success |
|
@martin-tarjanyi it seems the server closes the connection. Can you take a tcp dump then we will be able to see clearly what's happening. |
|
@martin-tarjanyi Were you able to obtain a tcp dump in order to check whether the server closes the connection? |
|
I'm facing the same exact issue. Tried reactor-netty v0.7.9 and v0.8.5 (spring webflux) but only in Windows and not in linux. I'll upload the logs soon. I've checked the logs and found that this happens when a old tcp connection is being used that was idle for some time. I'll upload the logs as soon as I can. |
|
@Kirupakaran Try to use Reactor Netty latest releases (for 0.8.x it is 0.8.9.RELEASE) |
violetagg
removed
the
for/user-attention
|
Closing the issue as there is no author response. |
|
@violetagg I'm sorry but my company prevented me from providing the tcp dump as it is a thirdparty service. If I can replicate it, I'll open a new issue with tcp dump. |
violetagg
added
the
for/user-attention
|
@violetagg I got the same My full error is as below, `2021-03-18 11:27:22.458 - - WARN [-,,,] 15296 --- [ctor-http-nio-6] i.n.c.AbstractChannelHandlerContext : An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.io.IOException: An existing connection was forcibly closed by the remote host I have configured my Webclient as below, ` @bean }` But when I remove the |
|
@hdineth Open a new issue and if possible switch to DEBUG. |
|
Hi @violetagg I'm not sure whether this has been fixed in latest RELEASE. My current version is 0.9.0.RELEASE. Find below DEBUG logs. |
|
I am getting this if I shut down connected reactor netty client |
|
I am getting this if I shut down connected client *************************************** Server Implementation ******************************************* *************************************** Client Implementation ******************************************* |
|
Hi Guys I am also struggling with this , and I read up the comments but I could not see the solution . Did anyone managed to explicitly close the webclient connection before its auto forced by remote server? |
|
Hey all, just dropping some details about my experience with this. There's a handful of issues that relate to this issue that I dug up over time in the reactor-netty repo here, I think I will try to go through them and link my comment for visibility. Issue Details Specifically, the problem I was running into was the application (which uses reactor-netty) was being routed through an Azure load balancer. The Azure load balancer has some questionable defaults where it:
My main issue is with #2. You can read more about it here in Azure where they talk about enabling this setting here but it seems this is not the default (legacy reasons? this feature wasn't available until about 2018 it seems). When this flag is disabled the behavior is as follows:
When this flag is enabled the behavior is as follows:
So really for us the problematic part was Azure never sending a close notification so the client never knew the connection was stale (also the blackhole behavior rather than just closing immediately wasn't great, we had application level retries that covered this but it produced a delay then in API calls). Resolution Use Reactor-Netty connection max idle time
You can configure the Reactor Netty Caveats Use TCP keep-alive + idle time options Caveats
For more information see the JDK change that added this here. If you look at the resolution this was only added for OSX and Linux. It is still not supported for Windows as of Java 17. You could enable TCP keep-alive generically (which is supported on all platforms even in Java 8) and then configure the Produce application level keep-alive messages What we did We ended up just using the max idle setting at the Reactor-Netty pool level and defaulting it to < 4 minutes (220 seconds) to cover our most common lowest denominator (Azure load balancers configured this way). Again, note if the remote side actually sends a close notification this generally is not a problem. However, there is probably some rarer race condition if the remote side hits the idle timeout right as the client side tries to use the connection. You can find some issue descriptions like this, this requires more exact timing though to reproduce/would be rarer although probably good to prevent by tossing these connections out pre-emptively. If possible though I would also suggest changing the Azure load balancer setting to enable the TCP RST behavior so proper close notifications come through. In my case this was not infrastructure I controlled and would have been larger in scope/higher risk to change that setting. |
This was referenced Jan 12, 2022
|
I'm a developer facing this issue in 2023. I can see how load balancers not sending RST packets can create an unpredictable behaviour. At this date Azure does present a setting to enable sending RST packets, but one would still have to rely on using keep alive probes to extend/reset the load balancer max idle time. But I'm wondering if netty could close and open a new channel instead of closing and returning back an exception... It does look like components that act as passthroughs do not send RST packets and I would argue that 99.9% of the developers that use netty are not aware that they could end up with this sort of unpredictable behaviour in the future if they plugin a load balancer. It would be a more resilient behaviour from netty if it could close and open a new connection in this specific case. |
|
@miguellgramacho96 , could you clarify if you suggest that the connection pool should automatically reopen a connection that is closed on idle timeout, or if you suggest to reopen a new connection and retry to send a request that could not be sent over a stale connection ? in the latter case, I think that the reactor-netty client already automatically retries the request with either a new connection or with an existing available one from the pool (but only if the request headers have not been sent). Note also that, as @ryanrupp said, the connection will hang for some time (like "5-20 seconds") before giving up on a stale connection, so the request will be retried but we can't avoid the delay. maybe @violetagg will want to add comments on this ? |
|
@pderop Hi, thanks for replying. And sorry for my late reply. In the meantime I have set the max idle time lower than the AWS Network Load Balancer to solve the issue. I was facing the following issue Faced this issue every time a new request triggered the reuse of a channel that stayed idle for more than the aws nlb max idle time, as far as I understood. So yes, that led me to think that reactor-netty was not being able to reuse the connection somehow, but I'm also not aware of the source code of reactor-netty, like the case you brought up of reusing an existing channel from the pool only if the request headers have not been sent... |
|
We have the same issue with the Azure load balancer when using WebClient. However, our challenge is that even when we select a short max idle time using ConnectionProvider, the pool always retains one idle connection. We observed this behavior through the metrics information. |
|
@fsilvaortiz This issue is closed. Please open a new issue for your use case and provide detailed description and how the issue can be reproduced. |
|
From my previous comment here
Re:
Then tuning TCP keep at the client level may be the preferred route. The commit from here can be adjusted now to mention this (will try to submit a PR when I get a chance). Thinking about this again because I'm trying to resolve an intermittent |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior
WebClient HTTP request succeeds after 5 minutes of idle time.
Actual behavior
WebClient HTTP request fails after 5 minutes of idle time.
If read timeout is set, then I get ReadTimeoutException.
If not set, I get the following exception:
Steps to reproduce
1, Start application: https://github.com/martin-tarjanyi/webclient-bug/blob/master/src/main/java/com/example/webclientbug/WebclientBugApplication.java
2, Call http://localhost:8080/get - request succeeds
3, Wait ~5 minutes
4, Call http://localhost:8080/get - request fails
Reactor Netty version
0.8.9-RELEASE
JVM version (e.g.
java -version)1.8.0_181
OS version (e.g.
uname -a)Windows 10
Related issue
https://stackoverflow.com/questions/55621800/springboot-webclient-throws-an-existing-connection-was-forcibly-closed-by-the-r
The text was updated successfully, but these errors were encountered: