java.io.IOException: Close org.eclipse.jetty.server.HttpConnection$SendCallback@5b27455[PROCESSING]

[chasingwind77]

We observed this exception in the log (with embedded Jetty version 9.4.14.v20181114). I have tried to google it but haven't found any useful information. Is it a known issue?

at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427)
at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:507)
at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:273)
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:923)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base/java.lang.Thread.run(Thread.java:834)
java.io.IOException: Close org.eclipse.jetty.server.HttpConnection$SendCallback@5b27455[PROCESSING][i=HTTP/1.1{s=200,h=4,cl=30578},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@18de03b] in state PROCESSING
at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427)
at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:507)
at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:273)
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:923)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base/java.lang.Thread.run(Thread.java:834)

[gregw]

This exception indicates that the connection was closed whilst the connection was still trying to write something. Does that tally with your application and scenario?

[chasingwind77]

The application wasn't shut down or restarted when this exception was thrown. So the connection doesn't seem to be closed by the application.

[joakime]

ok, what about the other way, the client closing the connection? (not the server)
and can you include the entire stacktrace, including the lines indicating where it was logged from and its message? (this smells like a debug/ignored logging output, but i can only confirm with the entire stacktrace)

[chasingwind77]

Sorry due to legal reason this is the most of the stack trace I could include. The exception is thrown from the handle method of an implementation of AbstractHandler. But I'm pretty sure there is no logic for closing the connection in our implementation.

[chasingwind77]

And I haven't found any evidence that the client is closing the connection. Normally when that happens our server would log an error.

[chasingwind77]

I noticed line 427 of IteratingCallback.java was changed from:
onCompleteFailure(new ClosedChannelException());
to:
onCompleteFailure(new IOException(failure));
between version 9.4.12.v20180830 and version 9.4.13.v20181111 (We are upgrading from 9.4.11.v20180605 to 9.4.14.v20181114). Could you double check why IOException is thrown rather than ClosedChannelException?

[joakime]

@gregw looks like the onCompleteFailure(new IOException(failure)) change was from commit 4a9265d (websocket backport commit)

[chasingwind77]

Any update on this? For now we are catching the IOException and ignoring it - probably not the right thing to do, but don't see other choice.

[joakime]

@gregw can you verify if the behavior change from the backport found in commit 4a9265d is valid for jetty-9.4.x?
Should that change in onCompleteFailure() be persisted in jetty-9.4.x? or just be present in jetty-10.0.x?

[gregw]

The exception was changed to IOException because the failure could be other things than a remote close, eg a timeout, OOM, etc. The previous code was loosing the reason for the failure.
So that change will not have changed any behaviour, only how that behaviour is reported.

Thus all we know at this time is that the end-point is being closed. We need more information to decide if that is abnormal or normal. Note that if you need an NDA to reveal more detail, that is part of our normal commercial support offering.

[chasingwind77]

I checked and we are not able to take commercial support at this time. I'll close the issue and will reach out when we are in a situation to get the commercial support.

[varmax2511]

Hi @joakime @gregw -- I am running into similar issue where the same error is output. Our application was in good health at the time of this exception and there was no connection closure by the client. Can someone suggest what can be the cause here?

.....
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at com.<hidden>.pic.commons.service.ZipBombPreventingFilter.handle(ZipBombPreventingFilter.java:78)
	at com.<hidden>.pic.commons.service.ZipBombPreventingFilter.doFilter(ZipBombPreventingFilter.java:54)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at com.<hidden>.pic.commons.service.ContentLengthFilter.handle(ContentLengthFilter.java:165)
	at com.<hidden>.pic.commons.service.ContentLengthFilter.doFilter(ContentLengthFilter.java:136)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319)
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at com.<hidden>.pic.commons.service.RequestIdFilter.doFilter(RequestIdFilter.java:97)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322)
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)
	at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.io.IOException: Close SendCallback@1e5b245f[PROCESSING][i=HTTP/1.1{s=202,h=7,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@109cc9e4] in state PROCESSING
	at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:449)
	at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:524)
	at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:378)
	at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
	at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1104)
	... 3 common frames omitted 

[joakime]

@varmax2511 Jetty 9 is now at End of Community Support.

• EoCS (End of Community Support) for Jetty 9 - June 2022 #7958
• EoCS (End of Community Support) for Jetty 10 / Jetty 11 - January 2024 #10485

You should be using a supported version of Jetty at this point in time.
That means Jetty 12.

And yes, you can run Jetty 12 with javax.servlet by using the ee8 environment on it.