Support for Logstash secure settings from Kubernetes Secrets using keystore

[kaisecheng]

This commit adds support for keystore to Logstash operator.

The key values in keystore are available to Logstash pipelines as environment variables, which can resolve by ${KEY} notation. The keystore can be password protected by setting an environment variable called LOGSTASH_KEYSTORE_PASS. The password is expected to be declared in the main container in env.

Sample config with customized keystore password

apiVersion: v1
kind: Secret
metadata:
  name: logstash-keystore-pass
stringData:
  LOGSTASH_KEYSTORE_PASS: changed
---
apiVersion: v1
kind: Secret
metadata:
  name: logstash-secure-settings
stringData:
  HELLO: Hallo
  WORLD: Welt
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  version: 8.8.0
  count: 1
  pipelines:
    - pipeline.id: main
      config.string: |-
        input { exec { command => 'uptime' interval => 10 } }
        filter {
          if ("${HELLO:}" != "") {
            mutate { add_tag => ["awesome"] }
          }
        }
  secureSettings:
    - secretName: logstash-secure-settings
  podTemplate:
    spec:
      containers:
        - name: logstash
          env:
            - name: LOGSTASH_KEYSTORE_PASS
              valueFrom:
                secretKeyRef:
                  name: logstash-keystore-pass
                  key: LOGSTASH_KEYSTORE_PASS

A known issue is that the keystore command logstash-keystore is very slow in proportion to the number of key values to add. In my local machine, adding 10 keys needs 6 minutes to start Logstsah.

Adding or updating key values in keystore triggers pod rotation, while deleting a key does not.

This commit adds e2e tests TestLogstashKeystoreWithoutPassword and TestLogstashKeystoreWithPassword

A follow-up issue for improving the keystore command performance

[robbavey]

One question about whether we need to set a default password for the keystore.

[robbavey]

Do we need to set a default value? The keystore should would work without a password (even though we don't recommend it), and I'm not sure what this buys us.

[kaisecheng]

Without the password in env, logstash-keystore prompts user to input [y/N] to continue.
So, we need to either give [N] to stdin or set the default password

[robbavey]

We should be able to use yes to send the input to stdin, so you could change the keystoreCommand to

keystoreCommand = "yes | /usr/share/logstash/bin/logstash-keystore"

to pipe y in

or

keystoreCommand = "yes n | /usr/share/logstash/bin/logstash-keystore"

to pipe n in.

I'm not sure it matters a huge amount, but it might be easier to document if we don't set a default that doesn't really do anything

[kaisecheng]

oh! I am such a fool :(
will change the command

[robbavey]

Not at all!

And thank you!

[robbavey]

I would add this as a technical preview NOTE, rather than in the main text, and add that the startup time increases linearly with the number of variables added.
Something like:

NOTE: For the technical preview, the use of settings in the Logstash keystore may impact startup time for Logstash Pods. Startup time will increase linearly for each entry added to the keystore, and this could extend startup time significantly.

[robbavey]

We should be able to use yes to send the input to stdin, so you could change the keystoreCommand to

keystoreCommand = "yes | /usr/share/logstash/bin/logstash-keystore"

to pipe y in

or

keystoreCommand = "yes n | /usr/share/logstash/bin/logstash-keystore"

to pipe n in.

I'm not sure it matters a huge amount, but it might be easier to document if we don't set a default that doesn't really do anything

[robbavey]

Code LGTM - couple of suggestions on doc.

[kaisecheng]

@pebrc @thbkrkr @barkbay This is ready for review

[pebrc]

LGTM I guess another alternative to allowing batch adding to the keystore would be to optimise the startup time of the keystore commend with JVM techniques like CDS or Graal's native image.

[kaisecheng]

@pebrc Could you merge the PR? I don't have the permission to do so.