Logstash don't start with 7.9.0-SNAPSHOT and default values

[jmlrt]

Following 7.x branch bump to 7.9.0-SNAPSHOT artifacts in #661, Logstash chart doesn't start anymore with the default values:

[2020-06-12T11:46:07,182][ERROR][logstash.monitoringextension.pipelineregisterhook] Failed to set up the metrics pipeline {:message=>"With the default xpack.monitoring.elasticsearch.username, xpack.monitoring.elasticsearch.password must be set", :backtrace=>["/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:207:in `validate_authentication!'", "/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:50:in `es_options_from_settings'", "/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:41:in `es_options_from_settings_or_modules'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:201:in `generate_pipeline_config'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:180:in `setup_metrics_pipeline'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:142:in `after_agent'", "org/logstash/execution/EventDispatcherExt.java:91:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:388:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:268:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[2020-06-12T11:46:07,185][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: With the default xpack.monitoring.elasticsearch.username, xpack.monitoring.elasticsearch.password must be set>, :backtrace=>["/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:207:in `validate_authentication!'", "/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:50:in `es_options_from_settings'", "/usr/share/logstash/x-pack/lib/helpers/elasticsearch_options.rb:41:in `es_options_from_settings_or_modules'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:201:in `generate_pipeline_config'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:180:in `setup_metrics_pipeline'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:142:in `after_agent'", "org/logstash/execution/EventDispatcherExt.java:91:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:388:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:268:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[2020-06-12T11:46:07,247][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

It seems that Logstash 7.9 now requires an Elasticsearch user/password when xpack.monitoring.enabled is enabled (enabled by default in basic Logstash Docker image) while Logstash < 7.9 was fine with an Elasticsearch with no authentication.

[jmlrt]

@elastic/logstash can you take a look at that issue? I reproduced it with docker.elastic.co/logstash/logstash:7.9.0-SNAPSHOT Docker image outside of Helm charts.

When running with xpack.monitoring.enabled (default config in Logstash Docker image), Logstash is failing with With the default xpack.monitoring.elasticsearch.username, xpack.monitoring.elasticsearch.password must be set error, while docker.elastic.co/logstash/logstash:7.8.0 with the same default config is OK.

Is it some new expected behavior?

[jsvd]

@colinsurprenant: this seems related to the new support for api keys in central management and monitoring done in elastic/logstash#11864

[colinsurprenant]

@jmlrt @jsvd I confirm this is a behaviour change in 7.9 - I added a check to make sure a password is explicitly set when either relying on the implicit default username or when setting an explicit username - in other words, when relying on basic auth a password must be set in 7.9.

I would have to dig back the reasoning for this but IIRC a user must have a password in ES and thus a passwordless basic auth config is invalid.

LMKWYT.

[jmlrt]

I added a check to make sure a password is explicitly set when either relying on the implicit default username or when setting an explicit username

@colinsurprenant, @jsvd I agree that a password should be explicitely set when setting an explicit username, however IMO users should keep the possibility to use xpack.monitoring without basic auth with default config, so either the password check shouldn't apply when using the implicit username or xpack.monitoring.elasticsearch.username shouldn't have any implicit default value.

An alternative if we think that xpack.monitoring should always require basic auth is to disable it in default config.

Do you want me to create a ticket in Logstash repo as it's not related to Helm charts but related to Logstash default config?

[colinsurprenant]

@jmlrt @jsvd

• Was the actual monitoring working in the docker image < 7.9 using the provided config where monitoring was enabled by default thus using the default username but no password ??? or was logstash just starting but reporting ES auth errors??

• I think the idea of having a default username is really silly but this has been the case for a while now. Maybe the check should be like this then: if no password is provided and no explicit username then no authentication should be used. if a password is provided without an explicit username then the implicit default username is used with that password.

• Also, why is xpack.monitoring enabled by default in the docker image while it is not enabled by default in the standard logstash distributions?

[jsvd]

@colinsurprenant up until 7.8.0 using monitoring didn't require setting a user or password, and none is sent if only xpack.monitoring.enabled: true is set in the logstash.yml:

POST /_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s HTTP/1.1
Connection: Keep-Alive
Content-Type: application/json
Content-Length: 1714
Host: localhost:9200
User-Agent: Manticore 0.6.4
Accept-Encoding: gzip,deflate

So this is a regresssion.

[colinsurprenant]

Make sense, let's fix it then.

[colinsurprenant]

Pushed elastic/logstash#12094

[colinsurprenant]

the fix was merged fro 7.9.0, let me know if it fixes on your end @jmlrt !

[jmlrt]

the fix was merged fro 7.9.0, let me know if it fixes on your end @jmlrt !

Logstash test is OK with the new 7.9.0-SNAPSHOT image => elastic+helm-charts+7.x+integration-logstash.

Thank you for fixing it ❤️