Skip to content
This repository has been archived by the owner on May 16, 2023. It is now read-only.

[logstash] mounting PEM certificate to connect to Elasticsearch via TLS isn't working #587

Closed
jmlrt opened this issue Apr 17, 2020 · 4 comments · Fixed by #392
Closed

[logstash] mounting PEM certificate to connect to Elasticsearch via TLS isn't working #587

jmlrt opened this issue Apr 17, 2020 · 4 comments · Fixed by #392
Assignees
@jmlrt
Labels
bug Something isn't working logstash

Comments

@jmlrt
Copy link
Member

jmlrt commented Apr 17, 2020 •
edited
Loading

Describe the bug:
When mounting a certificate in Logstash pod, Logstash don't seem to read the volume.

Steps to reproduce:

Deploy Logstash chart using the following value file:

values.yaml 
logstashConfig:
  logstash.yml: |
    http.host: 0.0.0.0
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'
    xpack.monitoring.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'
    xpack.monitoring.elasticsearch.hosts: ["https://security-master:9200"]
    xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/elastic-certificate.pem
logstashPipeline:
  uptime.conf: |
    input { exec { command => "uptime" interval => 30 } }
    output { elasticsearch {
      hosts => ["https://security-master:9200"]
      cacert => "/usr/share/logstash/config/certs/elastic-certificate.pem"
      user => '${ELASTICSEARCH_USERNAME}'
      password => '${ELASTICSEARCH_PASSWORD}'
      index => "logstash"
      }
    }
secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/logstash/config/certs

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password

Expected behavior:

elastic-certificate-pem certificate in mounting into the pod (/usr/share/logstash/config/certs/elastic-certificate-pem) and Logstash can use it to connect to Elasticsearch

Provide logs and/or server output (if relevant):

kubectl log helm-logstash-security-logstash-0
[2019-12-02T22:17:24,545][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>java.security.cert.CertificateParsingException: signed fields invalid, :backtrace=>["sun.security.x509.X509CertImpl.parse(sun/security/x509/X509CertImpl.java:1829)", "sun.security.x509.X509CertImpl.<init>(sun/security/x509/X509CertImpl.java:194)", "sun.security.provider.X509Factory.parseX509orPKCS7Cert(sun/security/provider/X509Factory.java:476)", "sun.security.provider.X509Factory.engineGenerateCertificates(sun/security/provider/X509Factory.java:361)", "java.security.cert.CertificateFactory.generateCertificates(java/security/cert/CertificateFactory.java:478)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.setup_trust_store(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:634)", "org.jruby.RubyIO.ensureYieldClose(org/jruby/RubyIO.java:1164)", "org.jruby.RubyIO.open(org/jruby/RubyIO.java:1158)", "org.jruby.RubyKernel.open(org/jruby/RubyKernel.java:320)", "org.jruby.RubyKernel$INVOKER$s$0$3$open.call(org/jruby/RubyKernel$INVOKER$s$0$3$open.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.setup_trust_store(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:633)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$setup_trust_store$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.ssl_socket_factory_from_options(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:621)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$ssl_socket_factory_from_options$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.pool_builder(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:397)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$pool_builder$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.pool(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:405)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:209)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_adapter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:282)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.RUBY$method$build_adapter$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.build_pool(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:286)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.RUBY$method$build_pool$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:64)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.create_http_client(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:103)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$create_http_client$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.build(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:99)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:355)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:195)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1800)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:467)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$maybe_setup_out_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:149)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:108)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"], :thread=>"#<Thread:0x617ec9ce run>"}
[2019-12-02T22:17:24,627][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

Any additional context:

Related to #392
@jmlrt jmlrt added the bug Something isn't working label Apr 17, 2020
This was referenced Apr 21, 2020
Merged
Closed
@xueshoulai
Copy link

same error message.
any work around solution to set up the ELK with TLS?

@AhmedSamirAhmed
Copy link
Contributor

I have managed to run Logstash with Elasticsearch security enabled with the following workaround:

  1. decode secret "elastic-certificate-pem".
  2. save the private key to tls.key and certificate to tls.crt.
  3. create a TLS secret with name logstash-certificate for example as the following
    kubectl create secret tls logstash-certificate --key="tls.key" --cert="tls.crt
  4. edit values.yaml as the following
    secretMounts:
      - name: logstash-certificate-pem
        secretName: logstash-certificate-pem
        path: /usr/share/logstash/config/certs
    logstashConfig:
      logstash.yml: |
        http.host: 0.0.0.0
        xpack.monitoring.enabled: true

    logstashPipeline:
      uptime.conf: |
        input { exec { command => "uptime" interval => 30 } }
        output { elasticsearch { hosts => ["https://security-master:9200"] user => "elastic" password => "password" index => "uptime" ssl_certificate_verification => false cacert => "/usr/share/logstash/config/certs/tls.crt"} }

@rhizoet
Copy link

rhizoet commented May 4, 2020

Thanks @AhmedSamirAhmed ,

worked for me. Helped me a lot. Should be mentioned in the Readme.

@jmlrt jmlrt mentioned this issue Jun 18, 2020
Open
@jmlrt jmlrt added the logstash label Jun 18, 2020
@jmlrt jmlrt self-assigned this Jun 18, 2020
@jmlrt
Copy link
Member Author

jmlrt commented Jun 18, 2020

@AhmedSamirAhmed using .crt certificate is successfull and I was able to merge #392.
Thanks for your help 👍

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jmlrt jmlrt

Labels
bug Something isn't working logstash
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[logstash] add security example jmlrt/helm-charts
4 participants
@jmlrt @AhmedSamirAhmed @xueshoulai @rhizoet