[2019-05-28T09:54:47,804][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] before test [2019-05-28T09:54:47,805][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] [CorruptedFileIT#testReplicaCorruption]: setting up test [2019-05-28T09:54:47,806][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] with seed [C9685F9729306CD1] using [0] dedicated masters, [2] (data) nodes and [1] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:54:47,812][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:47,812][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:47,814][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_t0], node ID [M6q98lWCSP6Jet1awV7TEA], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:47,814][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:47,814][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:47,814][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:47,817][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:47,833][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:47,849][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:47,853][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:47,853][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:47,854][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_t1], node ID [w_F5O094QwWQ5zOaUy2kBQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:47,855][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:47,855][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:47,855][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:47,855][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:47,870][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:47,886][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:47,890][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:47,890][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:47,891][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_tc2], node ID [kiB8ixMyTLGEm0CPSHhtfg], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:47,891][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:47,891][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:47,891][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:47,892][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:47,901][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:47,915][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:47,916][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#1]]] starting ... [2019-05-28T09:54:47,916][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#2]]] starting ... [2019-05-28T09:54:47,923][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#3]]] starting ... [2019-05-28T09:54:47,926][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#2]]] publish_address {127.0.0.1:36043}, bound_addresses {[::1]:43039}, {127.0.0.1:36043} [2019-05-28T09:54:47,929][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#1]]] publish_address {127.0.0.1:39865}, bound_addresses {[::1]:34399}, {127.0.0.1:39865} [2019-05-28T09:54:47,947][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#3]]] publish_address {127.0.0.1:45965}, bound_addresses {[::1]:34687}, {127.0.0.1:45965} [2019-05-28T09:54:47,994][INFO ][o.e.c.c.Coordinator ] [node_t0] setting initial configuration to VotingConfiguration{w_F5O094QwWQ5zOaUy2kBQ,M6q98lWCSP6Jet1awV7TEA} [2019-05-28T09:54:48,069][INFO ][o.e.c.s.MasterService ] [node_t0] elected-as-master ([2] nodes joined)[{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865} elect leader, {node_t1}{w_F5O094QwWQ5zOaUy2kBQ}{qWxoVKAtRCSOB8Gs1qqeVw}{127.0.0.1}{127.0.0.1:36043} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}]}, added {{node_t1}{w_F5O094QwWQ5zOaUy2kBQ}{qWxoVKAtRCSOB8Gs1qqeVw}{127.0.0.1}{127.0.0.1:36043},} [2019-05-28T09:54:48,075][INFO ][o.e.c.c.CoordinationState] [node_t0] cluster UUID set to [gMxoFwLgRLyfg7-1YZSiJQ] [2019-05-28T09:54:48,075][INFO ][o.e.c.c.CoordinationState] [node_t1] cluster UUID set to [gMxoFwLgRLyfg7-1YZSiJQ] [2019-05-28T09:54:48,078][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [], current [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}]}, added {{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865},}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}} [2019-05-28T09:54:48,078][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#2]]] started [2019-05-28T09:54:48,079][INFO ][o.e.c.s.ClusterApplierService] [node_t0] master node changed {previous [], current [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}]}, added {{node_t1}{w_F5O094QwWQ5zOaUy2kBQ}{qWxoVKAtRCSOB8Gs1qqeVw}{127.0.0.1}{127.0.0.1:36043},}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:54:48,079][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#1]]] started [2019-05-28T09:54:48,108][INFO ][o.e.g.GatewayService ] [node_t0] recovered [0] indices into cluster_state [2019-05-28T09:54:48,968][INFO ][o.e.c.s.MasterService ] [node_t0] node-join[{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{6lQ5Z8CZQYW3ttlfQiiIEw}{127.0.0.1}{127.0.0.1:45965} join existing leader], term: 1, version: 4, reason: added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{6lQ5Z8CZQYW3ttlfQiiIEw}{127.0.0.1}{127.0.0.1:45965},} [2019-05-28T09:54:48,973][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{6lQ5Z8CZQYW3ttlfQiiIEw}{127.0.0.1}{127.0.0.1:45965},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}} [2019-05-28T09:54:48,974][INFO ][o.e.c.s.ClusterApplierService] [node_tc2] master node changed {previous [], current [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}]}, added {{node_t1}{w_F5O094QwWQ5zOaUy2kBQ}{qWxoVKAtRCSOB8Gs1qqeVw}{127.0.0.1}{127.0.0.1:36043},{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}} [2019-05-28T09:54:48,976][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#3]]] started [2019-05-28T09:54:48,976][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:48,976][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:48,977][INFO ][o.e.c.s.ClusterApplierService] [node_t0] added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{6lQ5Z8CZQYW3ttlfQiiIEw}{127.0.0.1}{127.0.0.1:45965},}, term: 1, version: 4, reason: Publication{term=1, version=4} [2019-05-28T09:54:49,019][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:49,019][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:49,030][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t0] adding template [random_index_template] for index patterns [*] [2019-05-28T09:54:49,034][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] [CorruptedFileIT#testReplicaCorruption]: all set up test [2019-05-28T09:54:49,036][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,037][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_t0] [test] creating index, cause [api], templates [random_index_template], shards [7]/[1], mappings [] [2019-05-28T09:54:49,047][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,048][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,052][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,058][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,067][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,074][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,077][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,085][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,091][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,118][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,122][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,150][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,158][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,162][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,173][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,181][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,317][INFO ][o.e.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][5]] ...]). [2019-05-28T09:54:49,324][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:49,324][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:49,364][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] Index [471] docs async: [false] bulk: [true] partitions [2] [2019-05-28T09:54:49,370][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,371][INFO ][o.e.c.m.MetaDataMappingService] [node_t0] [test/6AeLqz_HTGev_imzg8ZT6g] create_mapping [type] [2019-05-28T09:54:49,392][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:49,395][INFO ][o.e.c.m.MetaDataMappingService] [node_t0] [test/6AeLqz_HTGev_imzg8ZT6g] update_mapping [type] [2019-05-28T09:54:49,407][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,382][INFO ][o.e.c.s.ClusterSettings ] [node_tc2] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:51,387][INFO ][o.e.c.s.ClusterSettings ] [node_t1] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:51,389][INFO ][o.e.c.s.ClusterSettings ] [node_t0] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:51,392][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] Stopping and resetting node [node_t0] [2019-05-28T09:54:51,393][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:51,400][INFO ][o.e.c.s.ClusterApplierService] [node_tc2] master node changed {previous [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}], current []}, term: 1, version: 17, reason: becoming candidate: onLeaderFailure [2019-05-28T09:54:51,400][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [{node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865}], current []}, term: 1, version: 17, reason: becoming candidate: onLeaderFailure [2019-05-28T09:54:51,401][WARN ][o.e.c.NodeConnectionsService] [node_t1] failed to connect to node {node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node_t0][127.0.0.1:39865] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[main/:?] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[main/:?] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_212] at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.addConnectListener(SocketChannelContext.java:82) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSocketChannel.addConnectListener(NioSocketChannel.java:77) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.MockNioTransport$MockSocketChannel.addConnectListener(MockNioTransport.java:304) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:325) ~[main/:?] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:292) ~[main/:?] at org.elasticsearch.test.transport.StubbableTransport.openConnection(StubbableTransport.java:151) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[main/:?] at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[main/:?] at org.elasticsearch.test.transport.StubbableConnectionManager.connectToNode(StubbableConnectionManager.java:116) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:344) ~[main/:?] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[main/:?] at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [main/:?] at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:116) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:97) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:43) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:406) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:423) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:260) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:153) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] ... 1 more [2019-05-28T09:54:51,441][WARN ][o.e.c.NodeConnectionsService] [node_tc2] failed to connect to node {node_t0}{M6q98lWCSP6Jet1awV7TEA}{1WMHNtp7QUWi-Dm-jR-sYQ}{127.0.0.1}{127.0.0.1:39865} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node_t0][127.0.0.1:39865] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[main/:?] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[main/:?] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_212] at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:97) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:43) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:406) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:423) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:260) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:153) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:116) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] ... 8 more [2019-05-28T09:54:51,521][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:51,521][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:51,524][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:51,524][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] Stopping and resetting node [node_t1] [2019-05-28T09:54:51,526][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:51,594][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:51,594][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:51,597][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:51,597][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] Stopping and resetting node [node_tc2] [2019-05-28T09:54:51,600][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:51,601][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:51,601][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:51,603][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:51,604][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] creating node [node_t1] [2019-05-28T09:54:51,609][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:51,609][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:51,625][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_t1], node ID [M6q98lWCSP6Jet1awV7TEA], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:51,625][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:51,625][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:51,625][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:51,625][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:51,626][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:51,626][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:51,626][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:51,649][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:51,662][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:51,662][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] creating node [node_t0] [2019-05-28T09:54:51,667][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:51,667][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:51,681][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_t0], node ID [w_F5O094QwWQ5zOaUy2kBQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:51,682][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:51,682][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:51,682][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:51,682][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:51,707][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:51,722][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:51,722][INFO ][o.e.t.InternalTestCluster] [testReplicaCorruption] creating node [node_tc2] [2019-05-28T09:54:51,727][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:51,728][INFO ][o.e.e.NodeEnvironment ] [testReplicaCorruption] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:51,728][INFO ][o.e.n.Node ] [testReplicaCorruption] node name [node_tc2], node ID [kiB8ixMyTLGEm0CPSHhtfg], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster] [2019-05-28T09:54:51,729][INFO ][o.e.n.Node ] [testReplicaCorruption] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:51,729][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:51,729][INFO ][o.e.n.Node ] [testReplicaCorruption] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:51,729][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:51,739][INFO ][o.e.d.DiscoveryModule ] [testReplicaCorruption] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:51,754][INFO ][o.e.n.Node ] [testReplicaCorruption] initialized [2019-05-28T09:54:51,755][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#4]]] starting ... [2019-05-28T09:54:51,755][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#6]]] starting ... [2019-05-28T09:54:51,755][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#5]]] starting ... [2019-05-28T09:54:51,770][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#6]]] publish_address {127.0.0.1:40175}, bound_addresses {[::1]:33395}, {127.0.0.1:40175} [2019-05-28T09:54:51,808][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#5]]] publish_address {127.0.0.1:38783}, bound_addresses {[::1]:32973}, {127.0.0.1:38783} [2019-05-28T09:54:51,809][INFO ][o.e.c.c.Coordinator ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#5]]] cluster UUID [gMxoFwLgRLyfg7-1YZSiJQ] [2019-05-28T09:54:51,817][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#4]]] publish_address {127.0.0.1:44755}, bound_addresses {[::1]:39987}, {127.0.0.1:44755} [2019-05-28T09:54:51,818][INFO ][o.e.c.c.Coordinator ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#4]]] cluster UUID [gMxoFwLgRLyfg7-1YZSiJQ] [2019-05-28T09:54:51,904][INFO ][o.e.c.s.MasterService ] [node_t1] elected-as-master ([1] nodes joined)[{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 18, reason: master node changed {previous [], current [{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}]} [2019-05-28T09:54:51,909][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [], current [{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}]}, term: 2, version: 18, reason: Publication{term=2, version=18} [2019-05-28T09:54:51,911][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#4]]] started [2019-05-28T09:54:51,912][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783} join existing leader], term: 2, version: 19, reason: added {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},} [2019-05-28T09:54:51,917][INFO ][o.e.c.s.ClusterApplierService] [node_t0] master node changed {previous [], current [{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}]}, added {{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755},}, term: 2, version: 19, reason: ApplyCommitRequest{term=2, version=19, sourceNode={node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}} [2019-05-28T09:54:51,918][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#5]]] started [2019-05-28T09:54:51,918][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},}, term: 2, version: 19, reason: Publication{term=2, version=19} [2019-05-28T09:54:51,920][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,921][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,924][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,925][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,925][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,928][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,929][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,929][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,931][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,932][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,932][INFO ][o.e.c.s.ClusterSettings ] [node_t0] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:51,933][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,934][INFO ][o.e.c.s.ClusterSettings ] [node_t1] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:51,934][INFO ][o.e.g.GatewayService ] [node_t1] recovered [1] indices into cluster_state [2019-05-28T09:54:51,935][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,935][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,944][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,947][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,950][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,962][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,964][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,974][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,986][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:51,991][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,044][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,048][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,094][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][3]] ...]). [2019-05-28T09:54:52,847][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{sGh9D4UGTXGySwmt6KjUTg}{127.0.0.1}{127.0.0.1:40175} join existing leader], term: 2, version: 26, reason: added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{sGh9D4UGTXGySwmt6KjUTg}{127.0.0.1}{127.0.0.1:40175},} [2019-05-28T09:54:52,857][INFO ][o.e.c.s.ClusterApplierService] [node_t0] added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{sGh9D4UGTXGySwmt6KjUTg}{127.0.0.1}{127.0.0.1:40175},}, term: 2, version: 26, reason: ApplyCommitRequest{term=2, version=26, sourceNode={node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}} [2019-05-28T09:54:52,858][INFO ][o.e.c.s.ClusterApplierService] [node_tc2] master node changed {previous [], current [{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}]}, added {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755},}, term: 2, version: 26, reason: ApplyCommitRequest{term=2, version=26, sourceNode={node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}} [2019-05-28T09:54:52,859][INFO ][o.e.c.s.ClusterSettings ] [node_tc2] updating [cluster.routing.allocation.enable] from [all] to [primaries] [2019-05-28T09:54:52,859][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-3933789171672388399]-HASH=[132767F3A9001]-cluster[T#6]]] started [2019-05-28T09:54:52,859][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_tc2}{kiB8ixMyTLGEm0CPSHhtfg}{sGh9D4UGTXGySwmt6KjUTg}{127.0.0.1}{127.0.0.1:40175},}, term: 2, version: 26, reason: Publication{term=2, version=26} [2019-05-28T09:54:52,859][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:52,860][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:52,889][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:52,889][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:52,909][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] no modules loaded [2019-05-28T09:54:52,910][INFO ][o.e.p.PluginsService ] [testReplicaCorruption] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:52,934][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,934][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,935][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,937][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,939][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,940][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,942][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:52,954][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5 on node node_t1 [2019-05-28T09:54:52,955][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/4/index/segments_5 on node node_t1 [2019-05-28T09:54:52,955][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5 on node node_t0 [2019-05-28T09:54:52,956][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/6/index/segments_5 on node node_t1 [2019-05-28T09:54:52,956][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5 on node node_t0 [2019-05-28T09:54:52,956][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5 on node node_t0 [2019-05-28T09:54:52,957][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] corrupting file /var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/0/index/segments_5 on node node_t1 [2019-05-28T09:54:52,963][INFO ][o.e.c.s.ClusterSettings ] [node_tc2] updating [cluster.routing.allocation.enable] from [primaries] to [all] [2019-05-28T09:54:52,964][INFO ][o.e.c.s.ClusterSettings ] [node_t0] updating [cluster.routing.allocation.enable] from [primaries] to [all] [2019-05-28T09:54:52,976][INFO ][o.e.c.s.ClusterSettings ] [node_t1] updating [cluster.routing.allocation.enable] from [primaries] to [all] [2019-05-28T09:54:53,003][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][2]: failed to list shard for shard_store on node [M6q98lWCSP6Jet1awV7TEA] org.elasticsearch.action.FailedNodeException: Failed node [M6q98lWCSP6Jet1awV7TEA] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) ~[main/:?] at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:704) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44755][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][2]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:54:53,012][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][1]: failed to list shard for shard_store on node [w_F5O094QwWQ5zOaUy2kBQ] org.elasticsearch.action.FailedNodeException: Failed node [w_F5O094QwWQ5zOaUy2kBQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:38783][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][1]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:53,025][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][4]: failed to list shard for shard_store on node [M6q98lWCSP6Jet1awV7TEA] org.elasticsearch.action.FailedNodeException: Failed node [M6q98lWCSP6Jet1awV7TEA] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) ~[main/:?] at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:704) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44755][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][4]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/4/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/4/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:54:53,025][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][6]: failed to list shard for shard_store on node [M6q98lWCSP6Jet1awV7TEA] org.elasticsearch.action.FailedNodeException: Failed node [M6q98lWCSP6Jet1awV7TEA] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) ~[main/:?] at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:704) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44755][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][6]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/6/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/6/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:54:53,026][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][3]: failed to list shard for shard_store on node [w_F5O094QwWQ5zOaUy2kBQ] org.elasticsearch.action.FailedNodeException: Failed node [w_F5O094QwWQ5zOaUy2kBQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:38783][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][3]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:53,029][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][5]: failed to list shard for shard_store on node [w_F5O094QwWQ5zOaUy2kBQ] org.elasticsearch.action.FailedNodeException: Failed node [w_F5O094QwWQ5zOaUy2kBQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:38783][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][5]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:53,043][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [M6q98lWCSP6Jet1awV7TEA] org.elasticsearch.action.FailedNodeException: Failed node [M6q98lWCSP6Jet1awV7TEA] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) ~[main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) ~[main/:?] at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:704) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44755][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/0/index/segments_5"))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: java.io.EOFException: read past EOF: SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/0/index/segments_5") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:454) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:153) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:54:53,052][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,056][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,058][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,063][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,062][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,073][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][3]: failed to list shard for shard_store on node [w_F5O094QwWQ5zOaUy2kBQ] org.elasticsearch.action.FailedNodeException: Failed node [w_F5O094QwWQ5zOaUy2kBQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:38783][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][3]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:53,074][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][1]: failed to list shard for shard_store on node [w_F5O094QwWQ5zOaUy2kBQ] org.elasticsearch.action.FailedNodeException: Failed node [w_F5O094QwWQ5zOaUy2kBQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:38783][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][1]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:53,081][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,081][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t1] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,086][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,082][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][2]: failed to list shard for shard_store on node [M6q98lWCSP6Jet1awV7TEA] org.elasticsearch.action.FailedNodeException: Failed node [M6q98lWCSP6Jet1awV7TEA] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) [main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) [main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) [main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) [main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) [main/:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) [main/:?] at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:704) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t1][127.0.0.1:44755][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][2]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/2/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:54:53,099][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,101][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/1/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,105][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t1] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/4/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/4/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,106][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/3/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,109][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,114][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:53,115][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t1] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/6/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/6/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,119][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t1] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/0/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/0/indices/6AeLqz_HTGev_imzg8ZT6g/0/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,163][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t0] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5")))) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:328) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:349) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:185) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$500(PeerRecoveryTargetService.java:85) [main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:665) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-002/data/nodes/1/indices/6AeLqz_HTGev_imzg8ZT6g/5/index/segments_5") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140) ~[lucene-test-framework-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:301) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] ... 22 more [2019-05-28T09:54:53,390][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][4]] ...]). [2019-05-28T09:54:53,407][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] [CorruptedFileIT#testReplicaCorruption]: cleaning up after test [2019-05-28T09:54:53,418][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_t1] [test/6AeLqz_HTGev_imzg8ZT6g] deleting index [2019-05-28T09:54:53,527][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] removing template [random_index_template] [2019-05-28T09:54:53,541][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:53,544][INFO ][o.e.c.s.MasterService ] [node_t1] node-left[{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783} disconnected], term: 2, version: 36, reason: removed {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},} [2019-05-28T09:54:53,545][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:53,545][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:53,547][INFO ][o.e.c.s.ClusterApplierService] [node_tc2] removed {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},}, term: 2, version: 36, reason: ApplyCommitRequest{term=2, version=36, sourceNode={node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}} [2019-05-28T09:54:53,548][INFO ][o.e.c.s.ClusterApplierService] [node_t1] removed {{node_t0}{w_F5O094QwWQ5zOaUy2kBQ}{TyrKa6zBT-OB3kmK3Oue2w}{127.0.0.1}{127.0.0.1:38783},}, term: 2, version: 36, reason: Publication{term=2, version=36} [2019-05-28T09:54:53,548][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:53,553][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:53,556][INFO ][o.e.c.s.ClusterApplierService] [node_tc2] master node changed {previous [{node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755}], current []}, term: 2, version: 36, reason: becoming candidate: onLeaderFailure [2019-05-28T09:54:53,556][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:53,556][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:53,559][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:53,560][WARN ][o.e.c.NodeConnectionsService] [node_tc2] failed to connect to node {node_t1}{M6q98lWCSP6Jet1awV7TEA}{TNIgJ4URTSa4kysndDfBUw}{127.0.0.1}{127.0.0.1:44755} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node_t1][127.0.0.1:44755] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[main/:?] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[main/:?] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_212] at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.addConnectListener(SocketChannelContext.java:82) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSocketChannel.addConnectListener(NioSocketChannel.java:77) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.MockNioTransport$MockSocketChannel.addConnectListener(MockNioTransport.java:304) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:325) ~[main/:?] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:292) ~[main/:?] at org.elasticsearch.test.transport.StubbableTransport.openConnection(StubbableTransport.java:151) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[main/:?] at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[main/:?] at org.elasticsearch.test.transport.StubbableConnectionManager.connectToNode(StubbableConnectionManager.java:116) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:344) ~[main/:?] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[main/:?] at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [main/:?] at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:116) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:97) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:43) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:406) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:423) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:260) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:153) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] ... 1 more [2019-05-28T09:54:53,563][INFO ][o.e.n.Node ] [testReplicaCorruption] stopping ... [2019-05-28T09:54:53,565][INFO ][o.e.n.Node ] [testReplicaCorruption] stopped [2019-05-28T09:54:53,565][INFO ][o.e.n.Node ] [testReplicaCorruption] closing ... [2019-05-28T09:54:53,567][INFO ][o.e.n.Node ] [testReplicaCorruption] closed [2019-05-28T09:54:53,568][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] [CorruptedFileIT#testReplicaCorruption]: cleaned up after test [2019-05-28T09:54:53,568][INFO ][o.e.i.s.CorruptedFileIT ] [testReplicaCorruption] after test [2019-05-28T09:54:53,571][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] before test [2019-05-28T09:54:53,572][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] [CorruptedFileIT#testCorruptFileAndRecover]: setting up test [2019-05-28T09:54:53,572][INFO ][o.e.t.InternalTestCluster] [testCorruptFileAndRecover] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster] with seed [97FBF3E0AE7BA0EE] using [0] dedicated masters, [3] (data) nodes and [1] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:54:53,577][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:53,577][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:53,578][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] node name [node_t0], node ID [o-6cglHxSDqycmweHU5QMg], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster] [2019-05-28T09:54:53,578][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:53,578][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:53,578][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:53,579][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:53,594][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileAndRecover] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:53,610][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] initialized [2019-05-28T09:54:53,620][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:53,620][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:53,622][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] node name [node_t1], node ID [VMurTVUOTTmgelXW-0MzrA], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster] [2019-05-28T09:54:53,622][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:53,622][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:53,622][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:53,622][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:53,648][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileAndRecover] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:53,669][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] initialized [2019-05-28T09:54:53,674][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:53,674][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:53,676][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] node name [node_t2], node ID [uQRoZVYsR4qOFFieMHqDnQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster] [2019-05-28T09:54:53,676][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:53,676][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:53,676][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:53,676][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:53,677][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:53,690][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileAndRecover] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:53,706][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] initialized [2019-05-28T09:54:53,712][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:53,712][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileAndRecover] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:53,713][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] node name [node_tc3], node ID [6YLyUEzdR_2rp2uOcipdUQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster] [2019-05-28T09:54:53,713][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:53,713][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:53,713][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:53,714][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:53,727][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileAndRecover] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:53,743][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] initialized [2019-05-28T09:54:53,749][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#1]]] starting ... [2019-05-28T09:54:53,755][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#2]]] starting ... [2019-05-28T09:54:53,759][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#3]]] starting ... [2019-05-28T09:54:53,760][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#2]]] publish_address {127.0.0.1:41879}, bound_addresses {[::1]:43639}, {127.0.0.1:41879} [2019-05-28T09:54:53,760][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#4]]] starting ... [2019-05-28T09:54:53,763][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#1]]] publish_address {127.0.0.1:42595}, bound_addresses {[::1]:40381}, {127.0.0.1:42595} [2019-05-28T09:54:53,766][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#4]]] publish_address {127.0.0.1:43301}, bound_addresses {[::1]:41921}, {127.0.0.1:43301} [2019-05-28T09:54:53,782][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#3]]] publish_address {127.0.0.1:38633}, bound_addresses {[::1]:46841}, {127.0.0.1:38633} [2019-05-28T09:54:53,817][INFO ][o.e.c.c.Coordinator ] [node_t1] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-node_t2,VMurTVUOTTmgelXW-0MzrA,o-6cglHxSDqycmweHU5QMg} [2019-05-28T09:54:53,840][INFO ][o.e.c.s.MasterService ] [node_t1] elected-as-master ([2] nodes joined)[{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595} elect leader, {node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}]}, added {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},} [2019-05-28T09:54:53,844][INFO ][o.e.c.c.CoordinationState] [node_t1] cluster UUID set to [oQzPQbEzQNOTqwckniNLWg] [2019-05-28T09:54:53,845][INFO ][o.e.c.c.CoordinationState] [node_t0] cluster UUID set to [oQzPQbEzQNOTqwckniNLWg] [2019-05-28T09:54:53,848][INFO ][o.e.c.s.ClusterApplierService] [node_t0] master node changed {previous [], current [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}]}, added {{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879},}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:53,849][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#1]]] started [2019-05-28T09:54:53,851][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [], current [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}]}, added {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:54:53,852][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#2]]] started [2019-05-28T09:54:53,860][INFO ][o.e.g.GatewayService ] [node_t1] recovered [0] indices into cluster_state [2019-05-28T09:54:53,861][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633} join existing leader], term: 1, version: 3, reason: added {{node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633},} [2019-05-28T09:54:53,868][INFO ][o.e.c.s.ClusterApplierService] [node_t2] master node changed {previous [], current [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}]}, added {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879},}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:53,870][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#3]]] started [2019-05-28T09:54:53,871][INFO ][o.e.c.s.ClusterApplierService] [node_t0] added {{node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633},}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:53,875][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633},}, term: 1, version: 3, reason: Publication{term=1, version=3} [2019-05-28T09:54:54,772][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_tc3}{6YLyUEzdR_2rp2uOcipdUQ}{9mJfJ7lXTW-NNBEtp8WWDw}{127.0.0.1}{127.0.0.1:43301} join existing leader], term: 1, version: 5, reason: added {{node_tc3}{6YLyUEzdR_2rp2uOcipdUQ}{9mJfJ7lXTW-NNBEtp8WWDw}{127.0.0.1}{127.0.0.1:43301},} [2019-05-28T09:54:54,777][INFO ][o.e.c.s.ClusterApplierService] [node_t2] added {{node_tc3}{6YLyUEzdR_2rp2uOcipdUQ}{9mJfJ7lXTW-NNBEtp8WWDw}{127.0.0.1}{127.0.0.1:43301},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:54,777][INFO ][o.e.c.s.ClusterApplierService] [node_t0] added {{node_tc3}{6YLyUEzdR_2rp2uOcipdUQ}{9mJfJ7lXTW-NNBEtp8WWDw}{127.0.0.1}{127.0.0.1:43301},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:54,778][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] master node changed {previous [], current [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}]}, added {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},{node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633},{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:54,783][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[-7495129008502497042]-HASH=[13277D6F39695]-cluster[T#4]]] started [2019-05-28T09:54:54,784][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:54,784][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:54,788][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_tc3}{6YLyUEzdR_2rp2uOcipdUQ}{9mJfJ7lXTW-NNBEtp8WWDw}{127.0.0.1}{127.0.0.1:43301},}, term: 1, version: 5, reason: Publication{term=1, version=5} [2019-05-28T09:54:54,813][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:54,813][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:54,864][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] adding template [random_index_template] for index patterns [*] [2019-05-28T09:54:54,882][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] [CorruptedFileIT#testCorruptFileAndRecover]: all set up test [2019-05-28T09:54:54,882][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] --> cluster has [3] data nodes, corrupted primary will be overwritten [2019-05-28T09:54:54,882][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] using custom data_path for index: [rPplShWkMN] [2019-05-28T09:54:54,884][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:54,885][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_t1] [test] creating index, cause [api], templates [random_index_template], shards [1]/[1], mappings [] [2019-05-28T09:54:54,900][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:54,902][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:54,946][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:54,950][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:54,999][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]). [2019-05-28T09:54:55,006][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,015][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,026][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:55,026][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:55,112][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] no modules loaded [2019-05-28T09:54:55,112][INFO ][o.e.p.PluginsService ] [testCorruptFileAndRecover] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:55,200][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] Index [123] docs async: [true] bulk: [false] [2019-05-28T09:54:55,219][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,220][INFO ][o.e.c.m.MetaDataMappingService] [node_t1] [test/HPsoNV0cSA-Wb8BHCmFtkg] create_mapping [type] [2019-05-28T09:54:55,251][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,506][INFO ][o.e.t.CorruptionUtils ] [testCorruptFileAndRecover] Corrupting file -- flipping at position 28 from ffffff81 to ffffff82 file: _0.si [2019-05-28T09:54:55,507][INFO ][o.e.t.CorruptionUtils ] [testCorruptFileAndRecover] Checksum before: [3676925954] after: [1453016370] checksum value after corruption: 3676925954] file: _0.si length: 393 [2019-05-28T09:54:55,507][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] --> [test][0], node[uQRoZVYsR4qOFFieMHqDnQ], [P], s[STARTED], a[id=M0rAl6tNSEefXDDmiaoEKg] corrupted [2019-05-28T09:54:55,509][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,579][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_t1] updating number_of_replicas to [2] for indices [test] [2019-05-28T09:54:55,581][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,584][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,592][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,601][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,604][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,649][WARN ][o.e.i.e.Engine ] [node_t2] [test][0] failed engine [recovery] org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,652][WARN ][o.e.i.c.IndicesClusterStateService] [node_t2] [[test][0]] marking and sending shard failed due to [shard failure, reason [recovery]] org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,669][WARN ][o.e.c.r.a.AllocationService] [node_t1] failing shard [failed shard, shard [test][0], node[uQRoZVYsR4qOFFieMHqDnQ], [P], s[STARTED], a[id=M0rAl6tNSEefXDDmiaoEKg], message [shard failure, reason [recovery]], failure [CorruptIndexException[file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si"))))]], markAsStale [true]] org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,691][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,693][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [o-6cglHxSDqycmweHU5QMg] org.elasticsearch.action.FailedNodeException: Failed node [o-6cglHxSDqycmweHU5QMg] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t0][127.0.0.1:42595][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: java.io.FileNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(ElasticsearchMockDirectoryWrapper(HybridDirectory@/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/0/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@d8e89d8))): files: [write.lock] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1189) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more [2019-05-28T09:54:55,696][WARN ][o.e.i.c.IndicesClusterStateService] [node_t0] [[test][0]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][0]: Recovery failed from {node_t2}{uQRoZVYsR4qOFFieMHqDnQ}{-5j_IAy9Q8WXyiYzIFmuEg}{127.0.0.1}{127.0.0.1:38633} into {node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [0] files with total size of [0b] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,713][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,713][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,723][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,726][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,718][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,737][INFO ][o.e.i.s.IndexShard ] [node_t1] [test][0] primary-replica resync completed with 0 operations [2019-05-28T09:54:55,765][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,767][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,771][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,773][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,779][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,796][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,801][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,803][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,806][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,811][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,825][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,827][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,884][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,886][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,893][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,896][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,915][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,918][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,943][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,945][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,951][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:55,953][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:55,987][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,000][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,029][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,031][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,035][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,041][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,044][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,046][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,052][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,054][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,063][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,065][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,078][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,080][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,083][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,085][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,088][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,090][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,093][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,095][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,098][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,101][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,104][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,106][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,110][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,111][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,122][WARN ][o.e.g.G.InternalReplicaShardAllocator] [node_t1] [test][0]: failed to list shard for shard_store on node [uQRoZVYsR4qOFFieMHqDnQ] org.elasticsearch.action.FailedNodeException: Failed node [uQRoZVYsR4qOFFieMHqDnQ] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:223) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$100(TransportNodesAction.java:142) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:198) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:999) ~[main/:?] at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:991) ~[main/:?] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:940) ~[main/:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[main/:?] at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:780) ~[main/:?] at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:241) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:398) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_t2][127.0.0.1:38633][internal:cluster/nodes/indices/shard/store[n]] Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:113) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] ... 1 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,134][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,137][WARN ][o.e.i.IndexSettings ] [node_t2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:56,140][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t2] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,200][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_t2] error while listing local files, recovering as if there are none org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=7ncoch580uupiipnrjv5y3ap0, got=7pheht3te8ldnow7z7kosexus (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkIndexHeaderID(CodecUtil.java:351) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:256) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=db297002 actual=71a9d563 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-003/data/nodes/2/indices/HPsoNV0cSA-Wb8BHCmFtkg/0/index/_0.si")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:183) ~[main/:?] at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:212) ~[main/:?] at org.elasticsearch.index.store.Store.access$300(Store.java:131) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:872) ~[main/:?] at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:805) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:298) ~[main/:?] at org.elasticsearch.index.store.Store.getMetadata(Store.java:263) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1193) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:127) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:60) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:246) ~[main/:?] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:242) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:54:56,375][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]). [2019-05-28T09:54:56,658][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_t1] [test/HPsoNV0cSA-Wb8BHCmFtkg] deleting index [2019-05-28T09:54:56,872][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] [CorruptedFileIT#testCorruptFileAndRecover]: cleaning up after test [2019-05-28T09:54:56,891][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] removing template [random_index_template] [2019-05-28T09:54:56,919][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopping ... [2019-05-28T09:54:56,927][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopped [2019-05-28T09:54:56,927][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closing ... [2019-05-28T09:54:56,927][INFO ][o.e.c.s.MasterService ] [node_t1] node-left[{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595} disconnected], term: 1, version: 22, reason: removed {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},} [2019-05-28T09:54:56,932][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] removed {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},}, term: 1, version: 22, reason: ApplyCommitRequest{term=1, version=22, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:56,933][INFO ][o.e.c.s.ClusterApplierService] [node_t2] removed {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},}, term: 1, version: 22, reason: ApplyCommitRequest{term=1, version=22, sourceNode={node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}} [2019-05-28T09:54:56,934][INFO ][o.e.c.s.ClusterApplierService] [node_t1] removed {{node_t0}{o-6cglHxSDqycmweHU5QMg}{qBzcRJtnSpmxu6URBpmbCQ}{127.0.0.1}{127.0.0.1:42595},}, term: 1, version: 22, reason: Publication{term=1, version=22} [2019-05-28T09:54:56,939][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closed [2019-05-28T09:54:56,943][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopping ... [2019-05-28T09:54:56,958][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] master node changed {previous [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}], current []}, term: 1, version: 22, reason: becoming candidate: onLeaderFailure [2019-05-28T09:54:56,958][INFO ][o.e.c.s.ClusterApplierService] [node_t2] master node changed {previous [{node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879}], current []}, term: 1, version: 22, reason: becoming candidate: onLeaderFailure [2019-05-28T09:54:56,959][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopped [2019-05-28T09:54:56,959][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closing ... [2019-05-28T09:54:56,958][WARN ][o.e.c.NodeConnectionsService] [node_tc3] failed to connect to node {node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node_t1][127.0.0.1:41879] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[main/:?] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[main/:?] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_212] at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:97) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:43) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:406) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:423) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:260) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:153) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:116) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] ... 8 more [2019-05-28T09:54:56,959][WARN ][o.e.c.NodeConnectionsService] [node_t2] failed to connect to node {node_t1}{VMurTVUOTTmgelXW-0MzrA}{5paxKmH_RgSe74JI-TG2yw}{127.0.0.1}{127.0.0.1:41879} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node_t1][127.0.0.1:41879] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[main/:?] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[main/:?] at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_212] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_212] at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:119) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:97) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:43) ~[framework-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:406) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:423) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:260) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:153) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:116) ~[elasticsearch-nio-7.1.1-SNAPSHOT.jar:7.1.1-SNAPSHOT] ... 8 more [2019-05-28T09:54:56,972][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closed [2019-05-28T09:54:56,978][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopping ... [2019-05-28T09:54:56,985][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopped [2019-05-28T09:54:56,985][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closing ... [2019-05-28T09:54:56,989][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closed [2019-05-28T09:54:56,996][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopping ... [2019-05-28T09:54:57,000][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] stopped [2019-05-28T09:54:57,000][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closing ... [2019-05-28T09:54:57,002][INFO ][o.e.n.Node ] [testCorruptFileAndRecover] closed [2019-05-28T09:54:57,002][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] [CorruptedFileIT#testCorruptFileAndRecover]: cleaned up after test [2019-05-28T09:54:57,002][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileAndRecover] after test [2019-05-28T09:54:57,005][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] before test [2019-05-28T09:54:57,005][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] [CorruptedFileIT#testCorruptionOnNetworkLayerFinalizingRecovery]: setting up test [2019-05-28T09:54:57,006][INFO ][o.e.t.InternalTestCluster] [testCorruptionOnNetworkLayerFinalizingRecovery] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster] with seed [3136D8EC2D026430] using [1] dedicated masters, [2] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:54:57,012][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:57,013][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:57,014][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] node name [node_tm0], node ID [eOd2GNQ2R_yCLvTU2ytccQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster] [2019-05-28T09:54:57,014][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:57,014][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:57,014][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:57,015][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,015][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:54:57,015][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:57,015][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:57,015][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:57,016][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,060][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayerFinalizingRecovery] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:57,086][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] initialized [2019-05-28T09:54:57,091][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:57,092][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:57,093][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] node name [node_td1], node ID [3iLTaXSsSwq4hDrrfh3y3w], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster] [2019-05-28T09:54:57,093][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:57,093][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:57,094][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:57,094][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,094][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:54:57,094][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:57,094][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:57,095][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,123][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayerFinalizingRecovery] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:57,169][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] initialized [2019-05-28T09:54:57,175][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:57,175][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayerFinalizingRecovery] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:57,176][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] node name [node_td2], node ID [0zfSwxORSFOBHBNsD-4EhA], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster] [2019-05-28T09:54:57,176][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:57,176][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:57,176][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:57,177][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:57,186][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,239][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayerFinalizingRecovery] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:57,254][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] initialized [2019-05-28T09:54:57,276][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#2]]] starting ... [2019-05-28T09:54:57,282][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#1]]] starting ... [2019-05-28T09:54:57,286][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#1]]] publish_address {127.0.0.1:44571}, bound_addresses {[::1]:32803}, {127.0.0.1:44571} [2019-05-28T09:54:57,287][INFO ][o.e.c.c.Coordinator ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#1]]] setting initial configuration to VotingConfiguration{eOd2GNQ2R_yCLvTU2ytccQ} [2019-05-28T09:54:57,294][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#3]]] starting ... [2019-05-28T09:54:57,388][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#2]]] publish_address {127.0.0.1:40963}, bound_addresses {[::1]:40345}, {127.0.0.1:40963} [2019-05-28T09:54:57,420][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#3]]] publish_address {127.0.0.1:43001}, bound_addresses {[::1]:34783}, {127.0.0.1:43001} [2019-05-28T09:54:57,487][INFO ][o.e.c.s.MasterService ] [node_tm0] elected-as-master ([1] nodes joined)[{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}]} [2019-05-28T09:54:57,490][INFO ][o.e.c.c.CoordinationState] [node_tm0] cluster UUID set to [s7SW-trZQhiI-7Yk4-bRow] [2019-05-28T09:54:57,493][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] master node changed {previous [], current [{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:54:57,499][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#1]]] started [2019-05-28T09:54:57,527][INFO ][o.e.g.GatewayService ] [node_tm0] recovered [0] indices into cluster_state [2019-05-28T09:54:57,545][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001} join existing leader], term: 1, version: 3, reason: added {{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001},} [2019-05-28T09:54:57,548][INFO ][o.e.c.s.ClusterApplierService] [node_td2] master node changed {previous [], current [{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}]}, added {{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571},}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}} [2019-05-28T09:54:57,553][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#3]]] started [2019-05-28T09:54:57,554][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001},}, term: 1, version: 3, reason: Publication{term=1, version=3} [2019-05-28T09:54:57,561][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963} join existing leader], term: 1, version: 4, reason: added {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},} [2019-05-28T09:54:57,583][INFO ][o.e.c.s.ClusterApplierService] [node_td1] master node changed {previous [], current [{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}]}, added {{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001},{node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}} [2019-05-28T09:54:57,588][INFO ][o.e.c.s.ClusterApplierService] [node_td2] added {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}} [2019-05-28T09:54:57,589][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[3546260265479857200]-HASH=[13278A39624B1]-cluster[T#2]]] started [2019-05-28T09:54:57,596][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},}, term: 1, version: 4, reason: Publication{term=1, version=4} [2019-05-28T09:54:57,603][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,603][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,639][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,639][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,651][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] adding template [random_index_template] for index patterns [*] [2019-05-28T09:54:57,661][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] [CorruptedFileIT#testCorruptionOnNetworkLayerFinalizingRecovery]: all set up test [2019-05-28T09:54:57,666][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] using custom data_path for index: [DUClUufSfT] [2019-05-28T09:54:57,666][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] no modules loaded [2019-05-28T09:54:57,666][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayerFinalizingRecovery] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,687][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_tm0] [test] creating index, cause [api], templates [random_index_template], shards [1]/[0], mappings [] [2019-05-28T09:54:57,733][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]). [2019-05-28T09:54:57,743][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_tm0] updating number_of_replicas to [1] for indices [test] [2019-05-28T09:54:57,798][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]). [2019-05-28T09:54:57,808][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] [CorruptedFileIT#testCorruptionOnNetworkLayerFinalizingRecovery]: cleaning up after test [2019-05-28T09:54:57,818][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_tm0] [test/thhJp0diRz2hACKnt8tVkw] deleting index [2019-05-28T09:54:57,825][INFO ][o.e.t.s.M.Listener ] [node_td2] [test][0] start check index [2019-05-28T09:54:57,826][INFO ][o.e.t.s.M.Listener ] [node_td2] [test][0] end check index [2019-05-28T09:54:57,827][INFO ][o.e.t.s.M.Listener ] [node_td1] [test][0] start check index [2019-05-28T09:54:57,828][INFO ][o.e.t.s.M.Listener ] [node_td1] [test][0] end check index [2019-05-28T09:54:57,838][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] removing template [random_index_template] [2019-05-28T09:54:57,876][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopping ... [2019-05-28T09:54:57,891][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963} disconnected], term: 1, version: 14, reason: removed {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},} [2019-05-28T09:54:57,891][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopped [2019-05-28T09:54:57,891][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closing ... [2019-05-28T09:54:57,904][INFO ][o.e.c.s.ClusterApplierService] [node_td2] removed {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},}, term: 1, version: 14, reason: ApplyCommitRequest{term=1, version=14, sourceNode={node_tm0}{eOd2GNQ2R_yCLvTU2ytccQ}{3whxpF3_Qj-LcyNov-_WWg}{127.0.0.1}{127.0.0.1:44571}} [2019-05-28T09:54:57,905][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closed [2019-05-28T09:54:57,906][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td1}{3iLTaXSsSwq4hDrrfh3y3w}{8QV0aaDwQQiRSWu4s_h7Xw}{127.0.0.1}{127.0.0.1:40963},}, term: 1, version: 14, reason: Publication{term=1, version=14} [2019-05-28T09:54:57,907][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopping ... [2019-05-28T09:54:57,909][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001} disconnected], term: 1, version: 15, reason: removed {{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001},} [2019-05-28T09:54:57,911][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td2}{0zfSwxORSFOBHBNsD-4EhA}{DaQ5EXZST7SLZuV7GD5hgw}{127.0.0.1}{127.0.0.1:43001},}, term: 1, version: 15, reason: Publication{term=1, version=15} [2019-05-28T09:54:57,912][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopped [2019-05-28T09:54:57,912][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closing ... [2019-05-28T09:54:57,913][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closed [2019-05-28T09:54:57,930][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopping ... [2019-05-28T09:54:57,932][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] stopped [2019-05-28T09:54:57,932][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closing ... [2019-05-28T09:54:57,934][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayerFinalizingRecovery] closed [2019-05-28T09:54:57,934][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] [CorruptedFileIT#testCorruptionOnNetworkLayerFinalizingRecovery]: cleaned up after test [2019-05-28T09:54:57,934][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayerFinalizingRecovery] after test [2019-05-28T09:54:57,940][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] before test [2019-05-28T09:54:57,940][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] [CorruptedFileIT#testCorruptPrimaryNoReplica]: setting up test [2019-05-28T09:54:57,941][INFO ][o.e.t.InternalTestCluster] [testCorruptPrimaryNoReplica] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster] with seed [667D3BA556C4BAB7] using [1] dedicated masters, [2] (data) nodes and [1] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:54:57,946][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:57,946][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:57,947][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] node name [node_tm0], node ID [rWLJB3txSI2xvYBTb3Lshw], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster] [2019-05-28T09:54:57,947][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:57,947][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:57,947][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:57,948][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:57,948][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:57,948][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:57,948][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:57,948][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:57,949][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:57,983][INFO ][o.e.d.DiscoveryModule ] [testCorruptPrimaryNoReplica] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:58,018][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] initialized [2019-05-28T09:54:58,022][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:58,022][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:58,026][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] node name [node_td1], node ID [Csf4t3LCSIW-air3i_wl3g], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster] [2019-05-28T09:54:58,026][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:58,026][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:58,026][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:58,026][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:58,027][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:58,047][INFO ][o.e.d.DiscoveryModule ] [testCorruptPrimaryNoReplica] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:58,064][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] initialized [2019-05-28T09:54:58,069][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:58,069][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:58,070][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] node name [node_td2], node ID [OloMVF5VROKZvNxG67r-GQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster] [2019-05-28T09:54:58,070][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:58,070][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:58,070][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:58,071][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:58,091][INFO ][o.e.d.DiscoveryModule ] [testCorruptPrimaryNoReplica] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:58,108][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] initialized [2019-05-28T09:54:58,114][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:54:58,114][INFO ][o.e.e.NodeEnvironment ] [testCorruptPrimaryNoReplica] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:54:58,115][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] node name [node_tc3], node ID [gGrm4lltSyqEPpf0gjpmXg], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster] [2019-05-28T09:54:58,115][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:54:58,115][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:54:58,115][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.index.mapper.MockFieldFilterPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:54:58,116][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:58,128][INFO ][o.e.d.DiscoveryModule ] [testCorruptPrimaryNoReplica] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:54:58,143][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] initialized [2019-05-28T09:54:58,144][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#1]]] starting ... [2019-05-28T09:54:58,145][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#2]]] starting ... [2019-05-28T09:54:58,151][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#3]]] starting ... [2019-05-28T09:54:58,163][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#4]]] starting ... [2019-05-28T09:54:58,176][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#3]]] publish_address {127.0.0.1:41191}, bound_addresses {[::1]:35147}, {127.0.0.1:41191} [2019-05-28T09:54:58,178][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#4]]] publish_address {127.0.0.1:46331}, bound_addresses {[::1]:33563}, {127.0.0.1:46331} [2019-05-28T09:54:58,182][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#1]]] publish_address {127.0.0.1:36611}, bound_addresses {[::1]:33979}, {127.0.0.1:36611} [2019-05-28T09:54:58,182][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#2]]] publish_address {127.0.0.1:32965}, bound_addresses {[::1]:42445}, {127.0.0.1:32965} [2019-05-28T09:54:58,190][INFO ][o.e.c.c.Coordinator ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#1]]] setting initial configuration to VotingConfiguration{rWLJB3txSI2xvYBTb3Lshw} [2019-05-28T09:54:58,208][INFO ][o.e.c.s.MasterService ] [node_tm0] elected-as-master ([1] nodes joined)[{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}]} [2019-05-28T09:54:58,236][INFO ][o.e.c.c.CoordinationState] [node_tm0] cluster UUID set to [GB2RqKvBQFuLOGhdMOI1uA] [2019-05-28T09:54:58,240][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] master node changed {previous [], current [{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:54:58,242][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#1]]] started [2019-05-28T09:54:58,247][INFO ][o.e.g.GatewayService ] [node_tm0] recovered [0] indices into cluster_state [2019-05-28T09:54:59,200][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965} join existing leader], term: 1, version: 3, reason: added {{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},} [2019-05-28T09:54:59,209][INFO ][o.e.c.s.ClusterApplierService] [node_td1] master node changed {previous [], current [{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}]}, added {{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611},}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,224][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#2]]] started [2019-05-28T09:54:59,226][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},}, term: 1, version: 3, reason: Publication{term=1, version=3} [2019-05-28T09:54:59,227][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191} join existing leader], term: 1, version: 4, reason: added {{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},} [2019-05-28T09:54:59,231][INFO ][o.e.c.s.ClusterApplierService] [node_td1] added {{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,239][INFO ][o.e.c.s.ClusterApplierService] [node_td2] master node changed {previous [], current [{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}]}, added {{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611},{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,272][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#3]]] started [2019-05-28T09:54:59,274][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},}, term: 1, version: 4, reason: Publication{term=1, version=4} [2019-05-28T09:54:59,276][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331} join existing leader], term: 1, version: 5, reason: added {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},} [2019-05-28T09:54:59,279][INFO ][o.e.c.s.ClusterApplierService] [node_td2] added {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,284][INFO ][o.e.c.s.ClusterApplierService] [node_td1] added {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,285][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] master node changed {previous [], current [{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}]}, added {{node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611},{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:54:59,304][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[7385124545268857527]-HASH=[13278DB54FA98]-cluster[T#4]]] started [2019-05-28T09:54:59,304][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 5, reason: Publication{term=1, version=5} [2019-05-28T09:54:59,305][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:59,305][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:59,397][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:59,397][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:59,427][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:59,427][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:59,470][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] adding template [random_index_template] for index patterns [*] [2019-05-28T09:54:59,481][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] [CorruptedFileIT#testCorruptPrimaryNoReplica]: all set up test [2019-05-28T09:54:59,481][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] using custom data_path for index: [nEysgCOpdD] [2019-05-28T09:54:59,484][WARN ][o.e.i.IndexSettings ] [node_tm0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,485][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_tm0] [test] creating index, cause [api], templates [random_index_template], shards [7]/[0], mappings [] [2019-05-28T09:54:59,490][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,490][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,492][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,494][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,496][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,506][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,522][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,528][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,534][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,579][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][6], [test][5], [test][0]] ...]). [2019-05-28T09:54:59,617][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] no modules loaded [2019-05-28T09:54:59,617][INFO ][o.e.p.PluginsService ] [testCorruptPrimaryNoReplica] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:54:59,697][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] Index [571] docs async: [false] bulk: [true] partitions [5] [2019-05-28T09:54:59,702][WARN ][o.e.i.IndexSettings ] [node_tm0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:54:59,704][INFO ][o.e.c.m.MetaDataMappingService] [node_tm0] [test/F49WHxwXQhe1gvb6JR_dHg] create_mapping [type] [2019-05-28T09:55:00,206][INFO ][o.e.t.CorruptionUtils ] [testCorruptPrimaryNoReplica] Corrupting file -- flipping at position 2248 from 0 to 1 file: _0.cfs [2019-05-28T09:55:00,207][INFO ][o.e.t.CorruptionUtils ] [testCorruptPrimaryNoReplica] Checksum before: [1788149269] after: [958074625] checksum value after corruption: 1788149269] file: _0.cfs length: 5230 [2019-05-28T09:55:00,208][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_tm0] updating number_of_replicas to [1] for indices [test] [2019-05-28T09:55:00,210][WARN ][o.e.i.IndexSettings ] [node_tm0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,298][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,305][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,314][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,328][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,374][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,382][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,386][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,467][WARN ][o.e.i.r.RecoverySourceHandler] [node_td1] [test][6][recover to node_td2] 6 Corrupted file detected name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0] checksum mismatch [2019-05-28T09:55:00,467][WARN ][o.e.i.e.Engine ] [node_td1] [test][6] failed engine [recovery] org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=tkm8t1 actual=186so7i (resource=name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.cfs)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:00,505][WARN ][o.e.i.c.IndicesClusterStateService] [node_td1] [[test][6]] marking and sending shard failed due to [shard failure, reason [recovery]] org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=tkm8t1 actual=186so7i (resource=name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.cfs)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:00,537][WARN ][o.e.i.c.IndicesClusterStateService] [node_td2] [[test][6]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][6]: Recovery failed from {node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965} into {node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_td1][127.0.0.1:32965][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [4] files with total size of [6.3kb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=tkm8t1 actual=186so7i (resource=name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.cfs)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more [2019-05-28T09:55:00,565][WARN ][o.e.c.r.a.AllocationService] [node_tm0] failing shard [failed shard, shard [test][6], node[Csf4t3LCSIW-air3i_wl3g], [P], s[STARTED], a[id=XrttlietRQu22c8Zpri3Qw], message [shard failure, reason [recovery]], failure [CorruptIndexException[checksum failed (hardware problem?) : expected=tkm8t1 actual=186so7i (resource=name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.cfs))]], markAsStale [true]] org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=tkm8t1 actual=186so7i (resource=name [_0.cfs], length [5230], checksum [tkm8t1], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.cfs)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:00,568][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[test][6], [test][6]] ...]). [2019-05-28T09:55:00,585][WARN ][o.e.i.IndexSettings ] [node_td1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,630][WARN ][o.e.i.IndexSettings ] [node_td2] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:00,728][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] [CorruptedFileIT#testCorruptPrimaryNoReplica]: cleaning up after test [2019-05-28T09:55:00,954][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_tm0] [test/F49WHxwXQhe1gvb6JR_dHg] deleting index [2019-05-28T09:55:01,072][WARN ][o.e.i.IndexSettings ] [node_tm0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:01,081][WARN ][o.e.i.IndexSettings ] [node_tm0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:01,085][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] removing template [random_index_template] [2019-05-28T09:55:01,126][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopping ... [2019-05-28T09:55:01,141][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331} disconnected], term: 1, version: 23, reason: removed {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},} [2019-05-28T09:55:01,143][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopped [2019-05-28T09:55:01,143][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closing ... [2019-05-28T09:55:01,145][INFO ][o.e.c.s.ClusterApplierService] [node_td2] removed {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 23, reason: ApplyCommitRequest{term=1, version=23, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:55:01,145][INFO ][o.e.c.s.ClusterApplierService] [node_td1] removed {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 23, reason: ApplyCommitRequest{term=1, version=23, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:55:01,149][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_tc3}{gGrm4lltSyqEPpf0gjpmXg}{ndbHolyBRAm8ArloG-f71w}{127.0.0.1}{127.0.0.1:46331},}, term: 1, version: 23, reason: Publication{term=1, version=23} [2019-05-28T09:55:01,169][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closed [2019-05-28T09:55:01,192][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopping ... [2019-05-28T09:55:01,198][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965} disconnected], term: 1, version: 24, reason: removed {{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},} [2019-05-28T09:55:01,201][INFO ][o.e.c.s.ClusterApplierService] [node_td2] removed {{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},}, term: 1, version: 24, reason: ApplyCommitRequest{term=1, version=24, sourceNode={node_tm0}{rWLJB3txSI2xvYBTb3Lshw}{DCgnTeiFRiifQ3cgwx7Z0Q}{127.0.0.1}{127.0.0.1:36611}} [2019-05-28T09:55:01,204][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopped [2019-05-28T09:55:01,204][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closing ... [2019-05-28T09:55:01,204][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td1}{Csf4t3LCSIW-air3i_wl3g}{YjPmbtvBT1CQc1h7DGYb_g}{127.0.0.1}{127.0.0.1:32965},}, term: 1, version: 24, reason: Publication{term=1, version=24} [2019-05-28T09:55:01,209][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closed [2019-05-28T09:55:01,227][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopping ... [2019-05-28T09:55:01,230][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191} disconnected], term: 1, version: 25, reason: removed {{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},} [2019-05-28T09:55:01,230][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopped [2019-05-28T09:55:01,230][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closing ... [2019-05-28T09:55:01,233][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closed [2019-05-28T09:55:01,234][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td2}{OloMVF5VROKZvNxG67r-GQ}{-MQN17FMRFO3uwKUhqyI1g}{127.0.0.1}{127.0.0.1:41191},}, term: 1, version: 25, reason: Publication{term=1, version=25} [2019-05-28T09:55:01,263][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopping ... [2019-05-28T09:55:01,265][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] stopped [2019-05-28T09:55:01,265][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closing ... [2019-05-28T09:55:01,286][INFO ][o.e.n.Node ] [testCorruptPrimaryNoReplica] closed [2019-05-28T09:55:01,288][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] [CorruptedFileIT#testCorruptPrimaryNoReplica]: cleaned up after test [2019-05-28T09:55:01,288][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptPrimaryNoReplica] after test [2019-05-28T09:55:01,300][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] before test [2019-05-28T09:55:01,301][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] [CorruptedFileIT#testCorruptionOnNetworkLayer]: setting up test [2019-05-28T09:55:01,302][INFO ][o.e.t.InternalTestCluster] [testCorruptionOnNetworkLayer] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster] with seed [502055EA9E27A2F8] using [1] dedicated masters, [1] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:55:01,315][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:01,315][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:01,316][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] node name [node_tm0], node ID [WT6xXHjRRjuKkg9l4dBRNQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster] [2019-05-28T09:55:01,320][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:01,320][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:01,320][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:01,321][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:01,322][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:01,322][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:01,322][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:01,322][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:01,322][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:01,405][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayer] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:01,471][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] initialized [2019-05-28T09:55:01,477][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:01,491][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:01,493][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] node name [node_td1], node ID [5imZxIvRRwK57gHiAK05rA], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster] [2019-05-28T09:55:01,493][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:01,493][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:01,493][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:01,495][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:01,496][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:01,496][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:01,496][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:01,496][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:01,496][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:01,543][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayer] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:01,618][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] initialized [2019-05-28T09:55:01,635][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#2]]] starting ... [2019-05-28T09:55:01,637][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#1]]] starting ... [2019-05-28T09:55:01,673][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#1]]] publish_address {127.0.0.1:46137}, bound_addresses {[::1]:35789}, {127.0.0.1:46137} [2019-05-28T09:55:01,692][INFO ][o.e.c.c.Coordinator ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#1]]] setting initial configuration to VotingConfiguration{WT6xXHjRRjuKkg9l4dBRNQ} [2019-05-28T09:55:01,765][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#2]]] publish_address {127.0.0.1:44695}, bound_addresses {[::1]:45385}, {127.0.0.1:44695} [2019-05-28T09:55:01,827][INFO ][o.e.c.s.MasterService ] [node_tm0] elected-as-master ([1] nodes joined)[{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}]} [2019-05-28T09:55:01,831][INFO ][o.e.c.c.CoordinationState] [node_tm0] cluster UUID set to [nDfY-UEqSbOtsqIQfqdC9A] [2019-05-28T09:55:01,835][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] master node changed {previous [], current [{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:55:01,842][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#1]]] started [2019-05-28T09:55:01,847][INFO ][o.e.g.GatewayService ] [node_tm0] recovered [0] indices into cluster_state [2019-05-28T09:55:01,866][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} join existing leader], term: 1, version: 3, reason: added {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},} [2019-05-28T09:55:01,881][INFO ][o.e.c.s.ClusterApplierService] [node_td1] master node changed {previous [], current [{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}]}, added {{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137},}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:01,886][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#2]]] started [2019-05-28T09:55:01,886][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:01,886][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:01,894][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},}, term: 1, version: 3, reason: Publication{term=1, version=3} [2019-05-28T09:55:01,976][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:01,976][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:02,050][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] adding template [random_index_template] for index patterns [*] [2019-05-28T09:55:02,058][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] [CorruptedFileIT#testCorruptionOnNetworkLayer]: all set up test [2019-05-28T09:55:02,058][INFO ][o.e.t.InternalTestCluster] [testCorruptionOnNetworkLayer] increasing cluster size from 1 to 2 [2019-05-28T09:55:02,064][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:02,064][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:02,065][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] node name [node_td2], node ID [GhwreASLTUSJOV4qz2u5ew], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster] [2019-05-28T09:55:02,065][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:02,066][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:02,066][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:02,066][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:02,123][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayer] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:02,151][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] initialized [2019-05-28T09:55:02,170][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#3]]] starting ... [2019-05-28T09:55:02,173][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#3]]] publish_address {127.0.0.1:46323}, bound_addresses {[::1]:42889}, {127.0.0.1:46323} [2019-05-28T09:55:02,191][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323} join existing leader], term: 1, version: 5, reason: added {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},} [2019-05-28T09:55:02,194][INFO ][o.e.c.s.ClusterApplierService] [node_td1] added {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:02,195][INFO ][o.e.c.s.ClusterApplierService] [node_td2] master node changed {previous [], current [{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}]}, added {{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137},{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:02,230][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#3]]] started [2019-05-28T09:55:02,230][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},}, term: 1, version: 5, reason: Publication{term=1, version=5} [2019-05-28T09:55:02,230][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:02,230][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:02,248][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:02,248][INFO ][o.e.e.NodeEnvironment ] [testCorruptionOnNetworkLayer] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:02,249][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] node name [node_td3], node ID [tKNRkoonR9ejQ5nZ2zPBqw], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster] [2019-05-28T09:55:02,249][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:02,249][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:02,249][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:02,250][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:02,272][INFO ][o.e.d.DiscoveryModule ] [testCorruptionOnNetworkLayer] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:02,302][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] initialized [2019-05-28T09:55:02,304][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#4]]] starting ... [2019-05-28T09:55:02,314][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#4]]] publish_address {127.0.0.1:40973}, bound_addresses {[::1]:42129}, {127.0.0.1:40973} [2019-05-28T09:55:02,353][INFO ][o.e.c.s.MasterService ] [node_tm0] node-join[{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} join existing leader], term: 1, version: 6, reason: added {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},} [2019-05-28T09:55:02,356][INFO ][o.e.c.s.ClusterApplierService] [node_td1] added {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:02,356][INFO ][o.e.c.s.ClusterApplierService] [node_td2] added {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:02,369][INFO ][o.e.c.s.ClusterApplierService] [node_td3] master node changed {previous [], current [{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}]}, added {{node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137},{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:02,417][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5773709188453081848]-HASH=[13279A3A5C791]-cluster[T#4]]] started [2019-05-28T09:55:02,420][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] added {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},}, term: 1, version: 6, reason: Publication{term=1, version=6} [2019-05-28T09:55:02,434][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_tm0] [test] creating index, cause [api], templates [random_index_template], shards [3]/[0], mappings [] [2019-05-28T09:55:02,513][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][1], [test][0]] ...]). [2019-05-28T09:55:02,531][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] no modules loaded [2019-05-28T09:55:02,531][INFO ][o.e.p.PluginsService ] [testCorruptionOnNetworkLayer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:02,578][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] Index [211] docs async: [true] bulk: [false] [2019-05-28T09:55:02,593][INFO ][o.e.c.m.MetaDataMappingService] [node_tm0] [test/Bfzd30h_Qh22MvYyjNLDcQ] create_mapping [type] [2019-05-28T09:55:03,226][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_tm0] updating number_of_replicas to [1] for indices [test] [2019-05-28T09:55:03,315][WARN ][o.e.i.r.RecoverySourceHandler] [node_td3] [test][2][recover to node_td1] 2 Remote file corruption on node {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695}, recovering name [_0.si], length [393], checksum [180g6i3], writtenBy [8.0.0]. local checksum OK org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=180g6i3 actual=wrcanc (resource=name [_0.si], length [393], checksum [180g6i3], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,319][WARN ][o.e.i.r.RecoverySourceHandler] [node_td3] [test][0][recover to node_td1] 0 Remote file corruption on node {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695}, recovering name [_1.si], length [393], checksum [ivq43s], writtenBy [8.0.0]. local checksum OK org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ivq43s actual=vq4vf3 (resource=name [_1.si], length [393], checksum [ivq43s], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_1.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,322][WARN ][o.e.i.c.IndicesClusterStateService] [node_td1] [[test][2]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][2]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [4] files with total size of [6.1kb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.transport.RemoteTransportException: [File corruption occurred on recovery but checksums are ok] Suppressed: org.elasticsearch.transport.RemoteTransportException: [node_td1][127.0.0.1:44695][internal:index/shard/recovery/file_chunk] Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=180g6i3 actual=wrcanc (resource=name [_0.si], length [393], checksum [180g6i3], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,356][WARN ][o.e.i.c.IndicesClusterStateService] [node_td1] [[test][0]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][0]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [16] files with total size of [21.2kb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.transport.RemoteTransportException: [File corruption occurred on recovery but checksums are ok] Suppressed: org.elasticsearch.transport.RemoteTransportException: [node_td1][127.0.0.1:44695][internal:index/shard/recovery/file_chunk] Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ivq43s actual=vq4vf3 (resource=name [_1.si], length [393], checksum [ivq43s], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_1.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,358][WARN ][o.e.c.r.a.AllocationService] [node_tm0] failing shard [failed shard, shard [test][2], node[5imZxIvRRwK57gHiAK05rA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=NdTaIWU7QVyLTtsCWnoUeg], unassigned_info[[reason=REPLICA_ADDED], at[2019-05-28T05:25:03.225Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[test][2]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695}]; nested: RemoteTransportException[[node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [4] files with total size of [6.1kb]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][2]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [4] files with total size of [6.1kb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.transport.RemoteTransportException: [File corruption occurred on recovery but checksums are ok] Suppressed: org.elasticsearch.transport.RemoteTransportException: [node_td1][127.0.0.1:44695][internal:index/shard/recovery/file_chunk] Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=180g6i3 actual=wrcanc (resource=name [_0.si], length [393], checksum [180g6i3], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_0.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,424][WARN ][o.e.c.r.a.AllocationService] [node_tm0] failing shard [failed shard, shard [test][0], node[5imZxIvRRwK57gHiAK05rA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=9IUZPLliTTudo9SxMxgMYg], unassigned_info[[reason=REPLICA_ADDED], at[2019-05-28T05:25:03.225Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[test][0]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695}]; nested: RemoteTransportException[[node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [16] files with total size of [21.2kb]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][0]: Recovery failed from {node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} into {node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$2(PeerRecoveryTargetService.java:253) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$1.handleException(PeerRecoveryTargetService.java:298) ~[main/:?] at org.elasticsearch.transport.PlainTransportFuture.handleException(PlainTransportFuture.java:97) ~[main/:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[main/:?] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.transport.RemoteTransportException: [node_td3][127.0.0.1:40973][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:182) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [16] files with total size of [21.2kb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:469) ~[main/:?] at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:180) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:106) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:54) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:112) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:109) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] ... 3 more Caused by: org.elasticsearch.transport.RemoteTransportException: [File corruption occurred on recovery but checksums are ok] Suppressed: org.elasticsearch.transport.RemoteTransportException: [node_td1][127.0.0.1:44695][internal:index/shard/recovery/file_chunk] Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ivq43s actual=vq4vf3 (resource=name [_1.si], length [393], checksum [ivq43s], writtenBy [8.0.0]) (resource=VerifyingIndexOutput(_1.si)) at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1237) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1215) ~[main/:?] at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1245) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:120) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.access$000(MultiFileWriter.java:43) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:200) ~[main/:?] at org.elasticsearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:68) ~[main/:?] at org.elasticsearch.indices.recovery.RecoveryTarget.writeFileChunk(RecoveryTarget.java:459) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:632) ~[main/:?] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$FileChunkTransportRequestHandler.messageReceived(PeerRecoveryTargetService.java:606) ~[main/:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[main/:?] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:03,679][INFO ][o.e.c.r.a.AllocationService] [node_tm0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]). [2019-05-28T09:55:03,942][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] [CorruptedFileIT#testCorruptionOnNetworkLayer]: cleaning up after test [2019-05-28T09:55:03,958][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_tm0] [test/Bfzd30h_Qh22MvYyjNLDcQ] deleting index [2019-05-28T09:55:04,027][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm0] removing template [random_index_template] [2019-05-28T09:55:04,072][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopping ... [2019-05-28T09:55:04,078][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695} disconnected], term: 1, version: 21, reason: removed {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},} [2019-05-28T09:55:04,079][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopped [2019-05-28T09:55:04,079][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closing ... [2019-05-28T09:55:04,082][INFO ][o.e.c.s.ClusterApplierService] [node_td2] removed {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},}, term: 1, version: 21, reason: ApplyCommitRequest{term=1, version=21, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:04,085][INFO ][o.e.c.s.ClusterApplierService] [node_td3] removed {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},}, term: 1, version: 21, reason: ApplyCommitRequest{term=1, version=21, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:04,090][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td1}{5imZxIvRRwK57gHiAK05rA}{L2LyA44WRd-khSPK6q9GtA}{127.0.0.1}{127.0.0.1:44695},}, term: 1, version: 21, reason: Publication{term=1, version=21} [2019-05-28T09:55:04,090][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closed [2019-05-28T09:55:04,098][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopping ... [2019-05-28T09:55:04,103][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323} disconnected], term: 1, version: 22, reason: removed {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},} [2019-05-28T09:55:04,106][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopped [2019-05-28T09:55:04,106][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closing ... [2019-05-28T09:55:04,108][INFO ][o.e.c.s.ClusterApplierService] [node_td3] removed {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},}, term: 1, version: 22, reason: ApplyCommitRequest{term=1, version=22, sourceNode={node_tm0}{WT6xXHjRRjuKkg9l4dBRNQ}{zL4d8cWUTv-dvGmdrblHFw}{127.0.0.1}{127.0.0.1:46137}} [2019-05-28T09:55:04,111][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closed [2019-05-28T09:55:04,112][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td2}{GhwreASLTUSJOV4qz2u5ew}{foP9teoYQQy4_HT5nEueJw}{127.0.0.1}{127.0.0.1:46323},}, term: 1, version: 22, reason: Publication{term=1, version=22} [2019-05-28T09:55:04,134][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopping ... [2019-05-28T09:55:04,137][INFO ][o.e.c.s.MasterService ] [node_tm0] node-left[{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973} disconnected], term: 1, version: 23, reason: removed {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},} [2019-05-28T09:55:04,137][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopped [2019-05-28T09:55:04,137][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closing ... [2019-05-28T09:55:04,139][INFO ][o.e.c.s.ClusterApplierService] [node_tm0] removed {{node_td3}{tKNRkoonR9ejQ5nZ2zPBqw}{soGftI71SROtQ1tiN4q68w}{127.0.0.1}{127.0.0.1:40973},}, term: 1, version: 23, reason: Publication{term=1, version=23} [2019-05-28T09:55:04,144][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closed [2019-05-28T09:55:04,147][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopping ... [2019-05-28T09:55:04,149][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] stopped [2019-05-28T09:55:04,149][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closing ... [2019-05-28T09:55:04,154][INFO ][o.e.n.Node ] [testCorruptionOnNetworkLayer] closed [2019-05-28T09:55:04,154][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] [CorruptedFileIT#testCorruptionOnNetworkLayer]: cleaned up after test [2019-05-28T09:55:04,154][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptionOnNetworkLayer] after test [2019-05-28T09:55:04,186][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] before test [2019-05-28T09:55:04,186][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] [CorruptedFileIT#testCorruptFileThenSnapshotAndRestore]: setting up test [2019-05-28T09:55:04,190][INFO ][o.e.t.InternalTestCluster] [testCorruptFileThenSnapshotAndRestore] Setup InternalTestCluster [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster] with seed [4F2F1D78CD2D9BD4] using [0] dedicated masters, [2] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed]) [2019-05-28T09:55:04,200][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileThenSnapshotAndRestore] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:04,200][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileThenSnapshotAndRestore] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:04,201][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] node name [node_t0], node ID [2sHjYLweRqOARAiT8UCe8g], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster] [2019-05-28T09:55:04,202][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:04,202][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:04,202][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] no modules loaded [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:04,202][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:04,250][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileThenSnapshotAndRestore] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:04,282][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] initialized [2019-05-28T09:55:04,287][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileThenSnapshotAndRestore] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.8tb], net total_space [1.8tb], types [xfs] [2019-05-28T09:55:04,287][INFO ][o.e.e.NodeEnvironment ] [testCorruptFileThenSnapshotAndRestore] heap size [491mb], compressed ordinary object pointers [true] [2019-05-28T09:55:04,292][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] node name [node_t1], node ID [eFrX4GI-QCmh6WEzwuVCnQ], cluster name [TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster] [2019-05-28T09:55:04,292][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] version[7.1.1], pid[2554033], build[unknown/unknown/Unknown/Unknown], OS[Linux/4.14.70-72.55.amzn2.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_212/25.212-b10] [2019-05-28T09:55:04,292][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] JVM home [/var/lib/jenkins/.java/oracle-8u212-linux/jre] [2019-05-28T09:55:04,292][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=12, -Des.scripting.update.ctx_in_params=false, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.2.1-all/bviwmvmbexq6idcscbicws5me/gradle-5.2.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/5.2.1/workerMain/gradle-worker.jar, -Djava.awt.headless=true, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Druntime.java=8, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=92FC281AD50DD500, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/heapdump, -esa, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] no modules loaded [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.MockIndexEventListener$TestPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] [2019-05-28T09:55:04,293][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:04,347][INFO ][o.e.d.DiscoveryModule ] [testCorruptFileThenSnapshotAndRestore] using discovery type [zen] and seed hosts providers [settings, file] [2019-05-28T09:55:04,369][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] initialized [2019-05-28T09:55:04,382][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#1]]] starting ... [2019-05-28T09:55:04,383][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#2]]] starting ... [2019-05-28T09:55:04,412][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#1]]] publish_address {127.0.0.1:42103}, bound_addresses {[::1]:33997}, {127.0.0.1:42103} [2019-05-28T09:55:04,417][INFO ][o.e.t.TransportService ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#2]]] publish_address {127.0.0.1:38895}, bound_addresses {[::1]:41871}, {127.0.0.1:38895} [2019-05-28T09:55:04,423][INFO ][o.e.c.c.Coordinator ] [node_t1] setting initial configuration to VotingConfiguration{2sHjYLweRqOARAiT8UCe8g,eFrX4GI-QCmh6WEzwuVCnQ} [2019-05-28T09:55:04,466][INFO ][o.e.c.s.MasterService ] [node_t1] elected-as-master ([2] nodes joined)[{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103} elect leader, {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]}, added {{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103},} [2019-05-28T09:55:04,472][INFO ][o.e.c.c.CoordinationState] [node_t1] cluster UUID set to [H__oTnzgT8izHgJB4H7_Iw] [2019-05-28T09:55:04,473][INFO ][o.e.c.c.CoordinationState] [node_t0] cluster UUID set to [H__oTnzgT8izHgJB4H7_Iw] [2019-05-28T09:55:04,476][INFO ][o.e.c.s.ClusterApplierService] [node_t0] master node changed {previous [], current [{node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]}, added {{node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895},}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}} [2019-05-28T09:55:04,476][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#1]]] started [2019-05-28T09:55:04,477][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [], current [{node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]}, added {{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103},}, term: 1, version: 1, reason: Publication{term=1, version=1} [2019-05-28T09:55:04,478][INFO ][o.e.n.Node ] [[test_TEST-CHILD_VM=[0]-CLUSTER_SEED=[5705811657577307092]-HASH=[1327A4FCDDCD2]-cluster[T#2]]] started [2019-05-28T09:55:04,498][INFO ][o.e.g.GatewayService ] [node_t1] recovered [0] indices into cluster_state [2019-05-28T09:55:04,510][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] adding template [random_index_template] for index patterns [*] [2019-05-28T09:55:04,529][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] [CorruptedFileIT#testCorruptFileThenSnapshotAndRestore]: all set up test [2019-05-28T09:55:04,529][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] no modules loaded [2019-05-28T09:55:04,529][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:04,554][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,555][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_t1] [test] creating index, cause [api], templates [random_index_template], shards [10]/[0], mappings [] [2019-05-28T09:55:04,566][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,568][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,574][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,585][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,604][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,612][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,614][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,629][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,655][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,662][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,721][WARN ][o.e.i.IndexSettings ] [node_t0] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,804][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,824][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][9]] ...]). [2019-05-28T09:55:04,832][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] no modules loaded [2019-05-28T09:55:04,832][INFO ][o.e.p.PluginsService ] [testCorruptFileThenSnapshotAndRestore] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] [2019-05-28T09:55:04,903][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] Index [1188] docs async: [false] bulk: [true] partitions [3] [2019-05-28T09:55:04,913][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,914][INFO ][o.e.c.m.MetaDataMappingService] [node_t1] [test/Gzuc2UCTRsmeeVzGh-sYOw] create_mapping [type] [2019-05-28T09:55:04,924][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:04,925][INFO ][o.e.c.m.MetaDataMappingService] [node_t1] [test/Gzuc2UCTRsmeeVzGh-sYOw] update_mapping [type] [2019-05-28T09:55:04,935][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:06,601][INFO ][o.e.t.CorruptionUtils ] [testCorruptFileThenSnapshotAndRestore] Corrupting file -- flipping at position 156 from 2 to 3 file: _1_2.fnm [2019-05-28T09:55:06,611][INFO ][o.e.t.CorruptionUtils ] [testCorruptFileThenSnapshotAndRestore] Checksum before: [2995402329] after: [3524445057] checksum value after corruption: 2995402329] file: _1_2.fnm length: 1128 [2019-05-28T09:55:06,611][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] --> shard [test][9], node[eFrX4GI-QCmh6WEzwuVCnQ], [P], s[STARTED], a[id=nxyUsvpnR2eBfu6xMiBd2g] has a corrupted file [2019-05-28T09:55:06,611][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] --> creating repository [2019-05-28T09:55:06,626][INFO ][o.e.r.RepositoriesService] [node_t1] put repository [test-repo] [2019-05-28T09:55:06,668][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] --> snapshot [2019-05-28T09:55:06,761][INFO ][o.e.s.SnapshotsService ] [node_t1] snapshot [test-repo:test-snap/4xiuwjPKSW-Fucq_dLftsA] started [2019-05-28T09:55:06,935][WARN ][o.e.i.e.Engine ] [node_t1] [test][9] failed engine [refresh failed source[version_table_flush]] java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) [main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) [main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) [main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) [main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) [main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) [main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) [main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:06,941][WARN ][o.e.i.c.IndicesClusterStateService] [node_t1] [[test][9]] marking and sending shard failed due to [shard failure, reason [refresh failed source[version_table_flush]]] java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:06,938][WARN ][o.e.s.SnapshotShardsService] [node_t1] [[test][9]][test-repo:test-snap/4xiuwjPKSW-Fucq_dLftsA] failed to snapshot shard org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: Failed to snapshot at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:378) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.engine.FlushFailedEngineException: Flush failed at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1719) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] ... 7 more Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1568) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] ... 7 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] ... 7 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) [main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:06,954][WARN ][o.e.c.r.a.AllocationService] [node_t1] failing shard [failed shard, shard [test][9], node[eFrX4GI-QCmh6WEzwuVCnQ], [P], s[STARTED], a[id=nxyUsvpnR2eBfu6xMiBd2g], message [shard failure, reason [refresh failed source[version_table_flush]]], failure [IOException[Invalid vInt detected (too many bits)]], markAsStale [true]] java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:138) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1553) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1714) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.acquireLastIndexCommit(InternalEngine.java:1942) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.acquireLastIndexCommit(IndexShard.java:1145) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:367) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:93) ~[main/:?] at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:314) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[main/:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:06,963][INFO ][o.e.c.r.a.AllocationService] [node_t1] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[test][9], [test][9]] ...]). [2019-05-28T09:55:06,965][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,002][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,067][WARN ][o.e.i.c.IndicesClusterStateService] [node_t1] [[test][9]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,075][WARN ][o.e.c.r.a.AllocationService] [node_t1] failing shard [failed shard, shard [test][9], node[eFrX4GI-QCmh6WEzwuVCnQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=nxyUsvpnR2eBfu6xMiBd2g], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-05-28T05:25:06.954Z], failed_attempts[1], delayed=false, details[failed shard on node [eFrX4GI-QCmh6WEzwuVCnQ]: shard failure, reason [refresh failed source[version_table_flush]], failure IOException[Invalid vInt detected (too many bits)]], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: IOException[Invalid vInt detected (too many bits)]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,077][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,117][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,158][WARN ][o.e.i.c.IndicesClusterStateService] [node_t1] [[test][9]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,160][WARN ][o.e.c.r.a.AllocationService] [node_t1] failing shard [failed shard, shard [test][9], node[eFrX4GI-QCmh6WEzwuVCnQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=nxyUsvpnR2eBfu6xMiBd2g], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-05-28T05:25:07.074Z], failed_attempts[2], delayed=false, details[failed shard on node [eFrX4GI-QCmh6WEzwuVCnQ]: failed recovery, failure RecoveryFailedException[[test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: IOException[Invalid vInt detected (too many bits)]; ], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: IOException[Invalid vInt detected (too many bits)]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,163][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,173][INFO ][o.e.s.SnapshotsService ] [node_t1] snapshot [test-repo:test-snap/4xiuwjPKSW-Fucq_dLftsA] completed with state [PARTIAL] [2019-05-28T09:55:07,185][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,212][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] --> snapshot terminated with state PARTIAL [2019-05-28T09:55:07,227][WARN ][o.e.i.c.IndicesClusterStateService] [node_t1] [[test][9]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,230][WARN ][o.e.c.r.a.AllocationService] [node_t1] failing shard [failed shard, shard [test][9], node[eFrX4GI-QCmh6WEzwuVCnQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=nxyUsvpnR2eBfu6xMiBd2g], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-05-28T05:25:07.160Z], failed_attempts[3], delayed=false, details[failed shard on node [eFrX4GI-QCmh6WEzwuVCnQ]: failed recovery, failure RecoveryFailedException[[test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: IOException[Invalid vInt detected (too many bits)]; ], allocation_status[no_valid_shard_copy]], message [failed recovery], failure [RecoveryFailedException[[test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: IOException[Invalid vInt detected (too many bits)]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,233][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,246][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] [CorruptedFileIT#testCorruptFileThenSnapshotAndRestore]: cleaning up after test [2019-05-28T09:55:07,273][WARN ][o.e.i.IndexSettings ] [node_t1] [test] [index.merge.enabled] is set to false, this should only be used in tests and can cause serious problems in production environments [2019-05-28T09:55:07,294][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_t1] [test/Gzuc2UCTRsmeeVzGh-sYOw] deleting index [2019-05-28T09:55:07,352][WARN ][o.e.i.c.IndicesClusterStateService] [node_t1] [[test][9]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test][9]: Recovery failed on {node_t1}{eFrX4GI-QCmh6WEzwuVCnQ}{AFrzt5tCQPW-7v5DNt_OfA}{127.0.0.1}{127.0.0.1:38895} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2344) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:438) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:204) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Caused by: java.io.IOException: Invalid vInt detected (too many bits) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:131) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] ... 4 more Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=b28a3659 actual=bfcf88ad (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(SimpleFSIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+7.1+multijob-unix-compatibility/os/amazon/server/build/testrun/integTest/temp/org.elasticsearch.index.store.CorruptedFileIT_92FC281AD50DD500-001/tempDir-007/data/nodes/1/indices/Gzuc2UCTRsmeeVzGh-sYOw/9/index/_1_2.fnm")))) at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:176) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:973) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:993) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.apache.lucene.index.IndexWriter.(IndexWriter.java:878) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55] at org.elasticsearch.index.engine.InternalEngine$AssertingIndexWriter.(InternalEngine.java:2604) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2125) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2115) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[main/:?] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:165) ~[main/:?] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1448) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1398) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[main/:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1687) ~[main/:?] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2340) ~[main/:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-05-28T09:55:07,448][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] removing template [random_index_template] [2019-05-28T09:55:07,454][INFO ][o.e.r.RepositoriesService] [node_t1] delete repository [test-repo] [2019-05-28T09:55:07,479][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] stopping ... [2019-05-28T09:55:07,483][INFO ][o.e.c.s.MasterService ] [node_t1] node-left[{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103} disconnected], term: 1, version: 39, reason: removed {{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103},} [2019-05-28T09:55:07,486][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] stopped [2019-05-28T09:55:07,486][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] closing ... [2019-05-28T09:55:07,488][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] closed [2019-05-28T09:55:07,489][INFO ][o.e.c.s.ClusterApplierService] [node_t1] removed {{node_t0}{2sHjYLweRqOARAiT8UCe8g}{ak3bXP-qSGmuXfy5XhLs3Q}{127.0.0.1}{127.0.0.1:42103},}, term: 1, version: 39, reason: Publication{term=1, version=39} [2019-05-28T09:55:07,491][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] stopping ... [2019-05-28T09:55:07,504][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] stopped [2019-05-28T09:55:07,504][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] closing ... [2019-05-28T09:55:07,506][INFO ][o.e.n.Node ] [testCorruptFileThenSnapshotAndRestore] closed [2019-05-28T09:55:07,506][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] [CorruptedFileIT#testCorruptFileThenSnapshotAndRestore]: cleaned up after test [2019-05-28T09:55:07,506][INFO ][o.e.i.s.CorruptedFileIT ] [testCorruptFileThenSnapshotAndRestore] after test