ERROR: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException

[divyankm]

I installed OS multinode using Helm having version: 2.6.0.

Data node is not getting added to the Opensearch Cluster: curl -XGET https://localhost:9200/_cat/nodes -u 'admin:admin' --insecure.

yaml files for data, client and master are attached.
opensearch-values-data.txt
opensearch-values-master.txt
opensearch-values-client.txt

Ref Link:1. https://opensearch.org/blog/setup-multinode-cluster-kubernetes/ 2.https://opensearch.org/docs/latest/install-and-configure/install-opensearch/helm/

Logs:

[eds@rnd-4 4px]$ kubectl get pods -n 4px
NAME                                         READY   STATUS    RESTARTS       AGE
opensearch-cluster-client-0                  1/1     Running   2 (134m ago)   17h
opensearch-cluster-data-0                    1/1     Running   0              133m
opensearch-cluster-master-0                  1/1     Running   0              133m

[eds@rnd-4 4px]$ kubectl logs opensearch-cluster-client-0 -n 4px | head -n 30
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init)
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.6.0
Detected OpenSearch Security Version: 2.6.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-11-01T03:56:49,527][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] version[2.6.0], pid[46], build[tar/7203a5af21a8a009aece1474446b437a3c674db6/2023-02-24T18:57:04.388618985Z], OS[Linux/5.14.0-162.18.1.el9_1.cloud.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-11-01T03:56:49,530][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]

[eds@rnd-4 4px]$ curl -XGET https://localhost:9200/_cat/nodes -u 'admin:admin' --insecure
10.244.5.49  51 56 0 0.20 0.26 0.31 m    master                                            * opensearch-cluster-master-0
10.244.1.205 32 60 0 1.82 1.92 1.96 dimr cluster_manager,data,ingest,remote_cluster_client - opensearch-cluster-client-0

[eds@rnd-4 4px]$ curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
{
  "name" : "opensearch-cluster-master-0",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "m6v71x5cQ6aGScL6rlo4wA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.6.0",
    "build_type" : "tar",
    "build_hash" : "7203a5af21a8a009aece1474446b437a3c674db6",
    "build_date" : "2023-02-24T18:57:04.388618985Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Error:

[eds@rnd-4 ~]$ kubectl logs opensearch-cluster-data-0 -n 4px | head -n 250
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init)
[2023-11-01T05:57:48,918][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-cluster-data-0] cluster-manager not discovered yet: have discovered [{opensearch-cluster-data-0}{5A7FCu3qRD6-IMNXGFX9Ig}{jAuWxXUZQSOFMCEtwjvSaw}{10.244.5.48}{10.244.5.48:9300}{di}{shard_indexing_pressure_enabled=true}]; discovery will continue using [10.244.5.49:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-11-01T05:57:49,051][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Wait for cluster to be available ...
[2023-11-01T05:57:49,137][WARN ][o.o.t.OutboundHandler    ] [opensearch-cluster-data-0] send message failed [channel: Netty4TcpChannel{localAddress=/10.244.5.48:44202, remoteAddress=opensearch-cluster-master-headless/10.244.5.49:9300}]
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1647) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1493) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
        ... 30 more
[2023-11-01T05:57:49,139][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-cluster-data-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Host/Environment (please complete the following information):

• Helm Version: 3.12.3
• Kubernetes Version: 1.27.4

Additional context
Add any other context about the problem here.

[prudhvigodithi]

[Untriage]
Adding @TheAlgo @andreasMore can you please add your thoughts on how to fix this?
@andreasMore #489 I see you also installed a seperate components and connected as a cluster.

[prudhvigodithi]

Hey @divyankm can you please confirm you have used the right roles? also please use cluster_manager instead of master.

[odinsy]

@prudhvigodithi your advice about using cluster_manager role is wrong, because of the chart is still using check for master

        {{- if (and (has "master" .Values.roles) (not .Values.singleNode)) }}
        - name: cluster.initial_master_nodes
          value: "{{ template "opensearch.endpoints" . }}"
        {{- end }}

https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch/templates/statefulset.yaml#L379