Issue with Kibana 8.11.2 and secure settings

[thbkrkr]

Issue

Kibana 8.11.2 introduced a breaking change that makes it unusable when configured with secure settings for all ECK versions.

See:

• kibana-keystore requires kibana.yml starting 8.11.2 kibana#172919

To reproduce:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: test
spec:
  secureSettings:
  - secretName: one-secure-settings-secret
  version: 8.11.2
  count: 1
  elasticsearchRef:
    name: test

Pod gets status Init:Error:

> k get po -l common.k8s.elastic.co/type=kibana
pod/n-kb-67b849f78f-2wtkz   0/1     Init:Error         2 (27s ago)     49s

Pod logs:

> k logs test-kb-67b849f78f-2wtkz -c elastic-internal-init-keystore
+ keystore_initialized_flag=/usr/share/kibana/config/elastic-internal-init-keystore.ok
+ [[ -f /usr/share/kibana/config/elastic-internal-init-keystore.ok ]]
+ echo 'Initializing keystore.'
+ /usr/share/kibana/bin/kibana-keystore create
Initializing keystore.
Error: ENOENT: no such file or directory, open '/usr/share/kibana/config/kibana.yml'
    at Object.openSync (node:fs:603:3)
    at readFileSync (node:fs:471:35)
    at readYaml (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:20:69)
    at getConfigFromFiles (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:56:18)
    at buildDataPaths (/usr/share/kibana/node_modules/@kbn/utils/src/path/index.js:41:82)
    at getDataPath (/usr/share/kibana/node_modules/@kbn/utils/src/path/index.js:71:36)
    at getKeystore (/usr/share/kibana/src/cli/keystore/get_keystore.js:21:63)
    at Object.<anonymous> (/usr/share/kibana/src/cli_keystore/cli_keystore.js:25:71)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1310:10)

Workaround

Explicitely mount kibana.yml from the elastic-internal-kibana-config volume:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: test
spec:
  secureSettings:
  - secretName: one-secure-settings-secret
  version: 8.11.2
  count: 1
  elasticsearchRef:
    name: test
  # --------------------------------------------------
  podTemplate:
    spec:
      initContainers:
      - name: elastic-internal-init-keystore
        volumeMounts:
        - mountPath: /usr/share/kibana/config/kibana.yml
          name: elastic-internal-kibana-config
          readonly: true
          subPath: kibana.yml
  # --------------------------------------------------

Long-term solution

Upgrade directly to 8.11.3 (which is not out as I write this but will be very soon).

[thbkrkr]

To create secure settings, ECK calls the Kibana keystore binary in an init container. Why there is no config.yaml in this init container? It's related to its volumes.

# pod volumes
{
  "emptyDir": {},
  "name": "elastic-internal-kibana-config-local"
},
{
  "name": "elastic-internal-kibana-config",
  "secret": {
    "defaultMode": 420,
    "optional": false,
    "secretName": "n-kb-config"
  }
}
# init container volumeMounts
 {
    "mountPath": "/mnt/elastic-internal/kibana-config",
    "name": "elastic-internal-kibana-config",
    "readOnly": true
  },
  {
    "mountPath": "/usr/share/kibana/config",
    "name": "elastic-internal-kibana-config-local"
  }

cloud-on-k8s/pkg/controller/kibana/config_reconcile.go

Lines 39 to 43 in 485d511

	// ConfigSharedVolume contains the Kibana config/ directory, it's an empty volume where the required configuration
	// is initialized by the elastic-internal-init-config init container. Its content is then shared by the init container
	// that creates the keystore and the main Kibana container.
	// This is needed in order to have in a same directory both the generated configuration and the keystore file which
	// is created in /usr/share/kibana/config since Kibana 7.9

More details:

• Add support for keystore with Kibana 7.9.0 #3566.