[elasticsearch] Fix bug in keystore initContainer

[ravishivt]

• Chart version not bumped (the versions are all bumped and released at the same time)
• README.md updated with any new values or changes
• Updated template tests in ${CHART}/tests/*.py
• Updated integration tests in ${CHART}/examples/*/test/goss.yaml

This fixes a bug in the keystore initContainer as reported in #280. The root cause was using set -u and the variable unset check [ ! -z "$ELASTIC_PASSWORD" ]. Doing the variable check this way violates set -u since it is considered to be accessing the variable. The fix is to change the unset check to [ ! -z ${ELASTIC_PASSWORD+x} ] as per https://stackoverflow.com/a/13864829/684893.

Note the error that the keystore initContainer reports without the fix:

k8s-prod$ kubectl logs test-elasticsearch-master-0 -c keystore
Created elasticsearch keystore in /usr/share/elasticsearch/config
Adding file /tmp/keystoreSecrets/aws-credentials/s3.client.default.access_key to keystore key s3.client.default.access_key
Adding file /tmp/keystoreSecrets/aws-credentials/s3.client.default.secret_key to keystore key s3.client.default.secret_key
sh: line 12: ELASTIC_PASSWORD: unbound variable

You can verify the issue and the fix using these simple tests:

$ sh -c 'set -u; [ ! -z "$DOG" ] && echo $DOG'
sh: DOG: unbound variable
$ sh -c 'set -u; [ ! -z ${DOG+x} ] && echo $DOG'
(no error)
$ sh -c 'set -u; DOG='ruff'; [ ! -z ${DOG+x} ] && echo $DOG'
ruff

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ravishivt]

fyi, I just signed the CLA.

[Crazybus]

LGTM! Thank you for finding and fixing this.

[Crazybus]

jenkins test this please

[dcvtruong]

@Crazybus Is the fixed available in a feature branch? I can give it a quick test run?

[Crazybus]

It has been merged into master. So you can try it out there. If you wait a day or two there should be a new release (no promises though).

[dcvtruong]

@Crazybus @ravishivt Giving the elastic pod is not accessible due to the privileged setting. How do I verify the slack url is set by the keystore? The elasticsearch deployment is 7.4.0.

This is from Kibana log when I try to create a watcher. The error shows the 'devopswatcher' is a invalid slack.

{"type":"error","@timestamp":"2019-10-04T02:56:12Z","tags":[],"pid":10,"level":"error","error":{"message":"[settings_exception] invalid slack [devopswatcher] account settings. missing required [secure_url] setting","name":"Error","stack":"[settings_exception] invalid slack [devopswatcher] account settings. missing required [secure_url] setting :: {\"path\":\"/_watcher/watch/ec63dc50-5f8f-4e5d-a6fd-bdcea5c16e97\",\"query\":{},\"body\":\"{\\\"metadata\\\":{\\\"xpack\\\":{\\\"type\\\":\\\"json\\\"},\\\"name\\\":\\\"k8s-alerts\\\"},\\\"trigger\\\":{\\\"schedule\\\":{\\\"interval\\\":\\\"5m\\\"}},\\\"input\\\":{\\\"search\\\":{\\\"request\\\":{\\\"indices\\\":[\\\"logstash*\\\"],\\\"body\\\":{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"match_phrase\\\":{\\\"log\\\":\\\"Failed to list *v1.*:\\\"}},{\\\"range\\\":{\\\"@timestamp\\\":{\\\"gte\\\":\\\"now-60m\\\",\\\"lte\\\":\\\"now\\\"}}}]}},\\\"_source\\\":[\\\"kubernetes.pod_name\\\",\\\"kubernetes.host\\\",\\\"log\\\"],\\\"sort\\\":[{\\\"@timestamp\\\":{\\\"order\\\":\\\"desc\\\"}}]}}}},\\\"condition\\\":{\\\"compare\\\":{\\\"ctx.payload.hits.total\\\":{\\\"gt\\\":0}}},\\\"throttle_period\\\":\\\"300m\\\",\\\"actions\\\":{\\\"notify-slack\\\":{\\\"throttle_period\\\":\\\"300m\\\",\\\"slack\\\":{\\\"account\\\":\\\"operations\\\",\\\"message\\\":{\\\"to\\\":[\\\"#devops-notifications\\\"],\\\"text\\\":\\\"MGMTDEV - Encountered  {{ctx.payload.hits.total}} Failed to list *v1 errors in the last hr.\\\",\\\"attachments\\\":[{\\\"title\\\":\\\"K8s Errors Found\\\",\\\"text\\\":\\\"{{ctx.payload.hits.hits}}\\\",\\\"color\\\":\\\"#ef0707\\\"}]}}}}}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"settings_exception\\\",\\\"reason\\\":\\\"invalid slack [devopswatcher] account settings. missing required [secure_url] setting\\\"}],\\\"type\\\":\\\"settings_exception\\\",\\\"reason\\\":\\\"invalid slack [devopswatcher] account settings. missing required [secure_url] setting\\\"},\\\"status\\\":500}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/watcher/watch/ec63dc50-5f8f-4e5d-a6fd-bdcea5c16e97","path":"/api/watcher/watch/ec63dc50-5f8f-4e5d-a6fd-bdcea5c16e97","href":"/api/watcher/watch/ec63dc50-5f8f-4e5d-a6fd-bdcea5c16e97"},"message":"[settings_exception] invalid slack [devopswatcher] account settings. missing required [secure_url] setting"}

$ helm get elasticsearch
REVISION: 3
RELEASED: Fri Oct  4 02:08:18 2019
CHART: elasticsearch-7.4.0
USER-SUPPLIED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: elasticsearch
esConfig:
  elasticsearch.yml: |
    xpack:
      security:
        enabled: false
    xpack.notification.slack:
      default_account: devopswatcher
      account:
        devopswatcher:
          message_defaults:
            from: elastic-config-slack2
esJavaOpts: -Xmx2g -Xms2g
esMajorVersion: ""
extraEnvs: []
extraInitContainers: ""
extraVolumeMounts: ""
extraVolumes: ""
fsGroup: ""
fullnameOverride: ""
httpPort: 9200
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.4.0
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources:
  limits:
    cpu: 25m
    memory: 128Mi
  requests:
    cpu: 25m
    memory: 128Mi
keystore:
- secretName: devopswatcher
labels: {}
lifecycle: {}
masterService: ""
masterTerminationFix: false
maxUnavailable: 1
minimumMasterNodes: 2
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity: {}
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
podSecurityPolicy:
  create: false
  name: ""
  spec:
    fsGroup:
      rule: RunAsAny
    privileged: true
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
    - secret
    - configMap
    - persistentVolumeClaim
priorityClassName: ""
protocol: http
rbac:
  create: false
  serviceAccountName: ""
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 3
resources:
  limits:
    cpu: 1000m
    memory: 5Gi
  requests:
    cpu: 100m
    memory: 2Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
schedulerName: ""
secretMounts: []
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service:
  annotations: {}
  httpPortName: http
  nodePort: 30998
  transportPortName: transport
  type: NodePort
sidecarResources:
  limits:
    cpu: 25m
    memory: 128Mi
  requests:
    cpu: 25m
    memory: 128Mi
sysctlInitContainer:
  enabled: true
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: elkdops05sc

COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: elasticsearch
esConfig:
  elasticsearch.yml: |
    xpack:
      security:
        enabled: false
    xpack.notification.slack:
      default_account: devopswatcher
      account:
        devopswatcher:
          message_defaults:
            from: elastic-config-slack2
esJavaOpts: -Xmx2g -Xms2g
esMajorVersion: ""
extraEnvs: []
extraInitContainers: ""
extraVolumeMounts: ""
extraVolumes: ""
fsGroup: ""
fullnameOverride: ""
httpPort: 9200
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.4.0
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources:
  limits:
    cpu: 25m
    memory: 128Mi
  requests:
    cpu: 25m
    memory: 128Mi
keystore:
- secretName: devopswatcher
labels: {}
lifecycle: {}
masterService: ""
masterTerminationFix: false
maxUnavailable: 1
minimumMasterNodes: 2
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity: {}
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
podSecurityPolicy:
  create: false
  name: ""
  spec:
    fsGroup:
      rule: RunAsAny
    privileged: true
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
    - secret
    - configMap
    - persistentVolumeClaim
priorityClassName: ""
protocol: http
rbac:
  create: false
  serviceAccountName: ""
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 3
resources:
  limits:
    cpu: 1000m
    memory: 5Gi
  requests:
    cpu: 100m
    memory: 2Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
schedulerName: ""
secretMounts: []
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service:
  annotations: {}
  httpPortName: http
  nodePort: 30998
  transportPortName: transport
  type: NodePort
sidecarResources:
  limits:
    cpu: 25m
    memory: 128Mi
  requests:
    cpu: 25m
    memory: 128Mi
sysctlInitContainer:
  enabled: true
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: elkdops05sc

HOOKS:
---
# elasticsearch-pcxur-test
apiVersion: v1
kind: Pod
metadata:
  name: "elasticsearch-pcxur-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "elasticsearch-sobwz-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.4.0"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "elasticsearch-master"
---
# Source: elasticsearch/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-master-config
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
data:
  elasticsearch.yml: |
    xpack:
      security:
        enabled: false
    xpack.notification.slack:
      default_account: devopswatcher
      account:
        devopswatcher:
          message_defaults:
            from: elastic-config-slack2
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    {}
    
spec:
  type: NodePort
  selector:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
    nodePort: 30998
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master-headless
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  # Create endpoints also if the related pod isn't ready
  publishNotReadyAddresses: true
  selector:
    app: "elasticsearch-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    esMajorVersion: "7"
spec:
  serviceName: elasticsearch-master-headless
  selector:
    matchLabels:
      app: "elasticsearch-master"
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: elkdops05sc
      
  template:
    metadata:
      name: "elasticsearch-master"
      labels:
        heritage: "Tiller"
        release: "elasticsearch"
        chart: "elasticsearch"
        app: "elasticsearch-master"
      annotations:
        
        configchecksum: 5093405c4357ef3e87189d682c66a9dbcc96c754cf90c46638d0cbf35726bba
    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
        
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "elasticsearch-master"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
        - name: esconfig
          configMap:
            name: elasticsearch-master-config
        - name: keystore
          emptyDir: {}
        - name: keystore-devopswatcher
          secret: 
            secretName: devopswatcher
            

      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.4.0"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          limits:
            cpu: 25m
            memory: 128Mi
          requests:
            cpu: 25m
            memory: 128Mi
          

      - name: keystore
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.4.0"
        command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
          set -euo pipefail

          elasticsearch-keystore create

          for i in /tmp/keystoreSecrets/*/*; do
            key=$(basename $i)
            echo "Adding file $i to keystore key $key"
            elasticsearch-keystore add-file "$key" "$i"
          done

          # Add the bootstrap password since otherwise the Elasticsearch entrypoint tries to do this on startup
          [ ! -z ${ELASTIC_PASSWORD+x} ] && echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x bootstrap.password

          cp -a /usr/share/elasticsearch/config/elasticsearch.keystore /tmp/keystore/
        env: 
          []
          
        resources: 
          limits:
            cpu: 25m
            memory: 128Mi
          requests:
            cpu: 25m
            memory: 128Mi
          
        volumeMounts:
          - name: keystore
            mountPath: /tmp/keystore
          - name: keystore-devopswatcher
            mountPath: /tmp/keystoreSecrets/devopswatcher

      containers:
      - name: "elasticsearch"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
          
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.4.0"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
          
          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 5Gi
          requests:
            cpu: 100m
            memory: 2Gi
          
        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,"
          - name: discovery.seed_hosts
            value: "elasticsearch-master-headless"
          - name: cluster.name
            value: "elasticsearch"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx2g -Xms2g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
        volumeMounts:
          - name: "elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data

          - name: keystore
            mountPath: /usr/share/elasticsearch/config/elasticsearch.keystore
            subPath: elasticsearch.keystore

          - name: esconfig
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml

Secrets:

$ kubectl get secrets devopswatcher -o yaml -n elk
apiVersion: v1
data:
  xpack.notification.slack.account.monitoring.secure_url: my_dirty_little_secrets...ferlifjwireot85aIJ
kind: Secret
metadata:
  creationTimestamp: "2019-10-04T02:06:40Z"
  name: devopswatcher
  namespace: elk
  resourceVersion: "2611751"
  selfLink: /api/v1/namespaces/elk/secrets/devopswatcher
  uid: 9a9c7646-e64b-11e9-9705-000d3a2b696b
type: Opaque

[Crazybus]

The account names don't match up. You added a secure_url for the account called monitoring but configured an account called devopswatcher. xpack.notification.slack.account.monitoring.secure_url should be xpack.notification.slack.account.devopswatcher.secure_url

data:
  xpack.notification.slack.account.monitoring.secure_url: my_dirty_little_secrets...ferlifjwireot85aIJ

    xpack.notification.slack:
      default_account: devopswatcher
      account:
        devopswatcher:

Please open an issue if you are still having problems. Comments on closed PRs might be missed.

[dcvtruong]

@Crazybus Oops on my part. Thanks for pointing out the incorrect setting.