Cross cluster search connection fails when the remote cluster has APM enabled and API key based connection is used

[rhr323]

Elasticsearch Version

8.14.2

Installed Plugins

No response

Java Version

22.0.1

OS Version

Linux ip-172-22-221-60 5.15.0-1035-aws #39-Ubuntu SMP Wed Apr 19 13:51:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Problem Description

When attempting to use Cross-Cluster Search (CCS) with API keys between two Elasticsearch clusters, the connection fails. The following exception is logged on the local cluster (the querying cluster):

Failure from _resolve/cluster lookup against cluster qa-overview: org.elasticsearch.transport.RemoteTransportException: [instance-0000000071][172.17.0.6:23217][cluster:internal/remote_cluster/handshake] 
Caused by: java.lang.IllegalArgumentException: Transport request header [tracestate] is not allowed for cross-cluster requests through the dedicated remote cluster server port

Connecting to another similar cluster from the same querying cluster works without issues. Based on observations, the main difference seems to be that the problematic remote cluster has APM enabled (e.g. telemetry.api_key entry in its keystore), which might be influencing the connection failure.

Steps to Reproduce

1. Set up two Elasticsearch clusters:
   • A query cluster that will initiate the connection for cross-cluster search.
   • A remote cluster to which the query cluster will connect.
2. Enable APM on the remote cluster
3. Configure the query cluster to use Cross-Cluster Search (CCS) with API keys to connect to the remote cluster.
4. Observe that the connection is not established, and the connection error is logged at the local cluster.

Logs (if relevant)

[instance-0000000001] Failure from _resolve/cluster lookup against cluster qa-overview: org.elasticsearch.transport.RemoteTransportException: [instance-0000000071][172.17.0.6:23217][cluster:internal/remote_cluster/handshake] Caused by: java.lang.IllegalArgumentException: Transport request header [tracestate] is not allowed for cross cluster requests through the dedicated remote cluster server port at org.elasticsearch.xpack.security.transport.CrossClusterAccessServerTransportFilter.validateHeaders(CrossClusterAccessServerTransportFilter.java:102) ~[?:?] at org.elasticsearch.xpack.security.transport.CrossClusterAccessServerTransportFilter.authenticate(CrossClusterAccessServerTransportFilter.java:87) ~[?:?] at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:642) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:273) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:96) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:821) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:124) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:96) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:61) ~[elasticsearch-8.15.0.jar:?] at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:48) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?]

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)