[CI] NetworkServiceTests.testPublishAnyLocalV4 failed on Java 14

[droberts195]

NetworkServiceTests.testPublishAnyLocalV4 failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/370/console

It is interesting that the runtime Java here is Java 14, which is not yet released, "Runtime JDK Version : 13 (Oracle Corporation 14-ea [OpenJDK 64-Bit Server VM 14-ea+25-1178])" to be precise.

The error was:

java.io.IOException: failed to check if interface [veth854822e] is up
Caused by: java.net.SocketException: No such device (getFlags() failed)
at java.net.NetworkInterface.isUp0(Native Method)
at java.net.NetworkInterface.isUp(NetworkInterface.java:458)
at org.elasticsearch.common.network.NetworkUtils.isUp(NetworkUtils.java:173)
at org.elasticsearch.common.network.NetworkUtils.filterAllAddresses(NetworkUtils.java:160)
at org.elasticsearch.common.network.NetworkUtils.getAllAddresses(NetworkUtils.java:201)
at org.elasticsearch.common.network.NetworkService.resolvePublishHostAddresses(NetworkService.java:155)
at org.elasticsearch.common.network.NetworkServiceTests.testPublishAnyLocalV4(NetworkServiceTests.java:108)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:832)

The REPRO command is:

./gradlew ':server:test' --tests "org.elasticsearch.common.network.NetworkServiceTests.testPublishAnyLocalV4" \
  -Dtests.seed=AA7FF538B7DFAC9D \
  -Dtests.security.manager=true \
  -Dtests.locale=ms-MY \
  -Dtests.timezone=Europe/Skopje \
  -Dcompiler.java=12

This doesn't reproduce using Java 12 as the runtime Java, so I am guessing it's related to Java 14. I'm not sure if this is something that needs to be fixed in the networking code, the Java 14 code or our CI worker setup, so pinging both teams initially for awareness.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Network)

[jaymode]

This looks similar to issues we've seen before with interfaces disappearing. #41758 was intended to fix it for most cases where we use localhost, however this test is trying to resolve all IPv4 addresses to figure out which address to list as the publish address.

[jaymode]

A similar failure happened in an intake build, https://gradle-enterprise.elastic.co/s/vw6wryxk4g464.

java.net.SocketException: No such device (getFlags() failed)
> No such device (getFlags() failed)

org.gradle.api.tasks.TaskExecutionException
: 
Execution failed for task ':x-pack:plugin:sql:qa:security:with-ssl:createNodeKeyStore'.
Open stacktrace
Caused by: 
org.gradle.api.UncheckedIOException
: 
java.net.SocketException: No such device (getFlags() failed)
Open stacktrace
Caused by: 
java.net.SocketException
: 
No such device (getFlags() failed)
Close stacktrace
at java.net.NetworkInterface.isUp0(Native Method)
at java.net.NetworkInterface.isUp(NetworkInterface.java:446)
at java_net_NetworkInterface$isUp$2.call(Unknown Source)
at SanEvaluator.getSubjectAlternativeNameString(/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part2/x-pack/plugin/sql/qa/security/with-ssl/build.gradle:216)
at SanEvaluator.toString(/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part2/x-pack/plugin/sql/qa/security/with-ssl/build.gradle:183)
at org.gradle.process.internal.DefaultExecHandleBuilder.getArgs(DefaultExecHandleBuilder.java:117)
at org.gradle.api.tasks.AbstractExecTask.getArgs(AbstractExecTask.java:121)
at org.elasticsearch.gradle.LoggedExec_Decorated.getArgs(Unknown Source)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:567)
at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$BeanPropertyValue$1$1.create(AbstractNestedRuntimeBeanNode.java:80)
at org.gradle.util.SingleMessageLogger.whileDisabled(SingleMessageLogger.java:507)
at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$BeanPropertyValue$1.get(AbstractNestedRuntimeBeanNode.java:76)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167)
at org.gradle.api.internal.tasks.properties.bean.AbstractNestedRuntimeBeanNode$BeanPropertyValue.call(AbstractNestedRuntimeBeanNode.java:148)
at org.gradle.api.internal.tasks.properties.AbstractValidatingProperty.validate(AbstractValidatingProperty.java:39)
at org.gradle.api.internal.tasks.properties.DefaultTaskProperties.validate(DefaultTaskProperties.java:193)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.validate(ExecuteActionsTaskExecuter.java:476)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:50)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)
at java.util.Optional.orElseGet(Optional.java:362)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)
at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:174)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:166)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:374)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:361)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:354)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:340)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.lang.Thread.run(Thread.java:830)