High I/O dispatcher thread when using RestHighLevelClient (elasticsearch-rest-high-level-client 7.6.2)

[sgooty]

Hi ,

We are see High I/O dispatcher thread running threads on JVM. These thread will never get killed and create massive count on JVM.

OS Details:
NAME="Oracle Linux Server"
VERSION="7.4"
ID="ol"
VERSION_ID="7.4"
PRETTY_NAME="Oracle Linux Server 7.4"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:4:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.4
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.4

JDK Details:
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

Tomcat
Server version: Apache Tomcat/9.0.17
Server built: Mar 13 2019 15:55:27 UTC
Server number: 9.0.17.0
OS Name: Linux
OS Version: 4.14.35-1902.301.1.el7uek.x86_64
Architecture: amd64
JVM Version: 1.8.0_262-b10
JVM Vendor: Oracle Corporation

Sample code :
RestClientBuilder builder = RestClient.builder(httpHosts.toArray(new HttpHost[httpHosts.size()])).setRequestConfigCallback(
new RestClientBuilder.RequestConfigCallback() {
@OverRide
public RequestConfig.Builder customizeRequestConfig(
RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder
.setConnectTimeout(propConnectTimeout)
.setSocketTimeout(propReadTimeout)
.setConnectionRequestTimeout(-1);
}
}).setHttpClientConfigCallback(new HttpClientConfigCallback() {
@OverRide
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setDefaultIOReactorConfig(
IOReactorConfig.custom()
.setIoThreadCount(1)
.build());
}
});

	RestHighLevelClient client = new RestHighLevelClient(builder);

Researched a quite but no luck and we are planning to implement this at bigger picture but this issue is killing all our q4 development goals.

PS: its using httpcore 4.4.12

[chefhub-afk]

Option #1:
upgrade spring-data-elasticsearch to 4.0.1 or above

Option #2:
httpasyncclient.jar to latest version

Option #3 - upgrade linux kernel (depending on the Linux version execute instructions)
yum -y update yum rpm
yum -y clean all
yum -y update

Option #4
Change this value to 3 seconds (3000) in the above code
.setConnectionRequestTimeout(-1);

Option #5:
I think you need to shutdown the connectionManager
https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/apidocs/org/apache/http/impl/nio/conn/PoolingNHttpClientConnectionManager.html#shutdown()

Once you try these options share updates

[sgooty]

Option #1:
upgrade spring-data-elasticsearch to 4.0.1 or above

Option #2:
httpasyncclient.jar to latest version

Option #3 - upgrade linux kernel (depending on the Linux version execute instructions)
yum -y update yum rpm
yum -y clean all
yum -y update

Option #4
Change this value to 3 seconds (3000) in the above code
.setConnectionRequestTimeout(-1);

Option #5:
I think you need to shutdown the connectionManager
https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/apidocs/org/apache/http/impl/nio/conn/PoolingNHttpClientConnectionManager.html#shutdown()

Once you try these options share updates

Option #1 not using Spring data
Option #2 using latest

org.apache.httpcomponents
httpasyncclient
4.1.4

Option #3 We are already and lastest.
Option #4 made this change still no luck.
Option #5: can you please provide more context.

RestClientBuilder builder = RestClient.builder(httpHosts.toArray(new HttpHost[httpHosts.size()])).setRequestConfigCallback(
new RestClientBuilder.RequestConfigCallback() {
@OverRide
public RequestConfig.Builder customizeRequestConfig(
RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder
.setConnectTimeout(10000)
.setSocketTimeout(450000)
.setConnectionRequestTimeout(-1);
}
}).setHttpClientConfigCallback(new HttpClientConfigCallback() {
@OverRide
public HttpAsyncClientBuilder customizeHttpClient(
HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setDefaultIOReactorConfig(
IOReactorConfig.custom()
.setIoThreadCount(1)
.setSoTimeout(500)
.setSoReuseAddress(true)
.build()).setConnectionManagerShared(true)
.setMaxConnTotal(200);
}
});

[original-brownbear]

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds of things to the Elasticsearch forum. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.

There's an active community in the forum that should be able to help get an answer to your question. As such, I hope you don't mind that I close this.