Fixup security install docs #205

Crazybus · 2019-07-05T10:26:00Z

Fixes: #193

The default makefile target was switched at some point in time to also
include installing secrets from our secret service. This obviously
doesn't work so good for users.

I took this as an opportunity to also pick up #27 by making sure that all secrets are generated fresh for each CI run. By using the same target for CI as the user documentation we can be sure that the docs won't go out of date again.

Crazybus · 2019-07-08T12:58:53Z

Thanks for testing it out. I'm going to pick up #27 and make sure that all secrets are generated on the fly during the automated testing. This will make sure that the examples will work for CI and for users.

Fixes: #193 The default makefile target was switched at some point in time to also include installing secrets from our secret service. This obviously doesn't work so good for users.

Closes: #27 This makes it possible to run all of the examples without needing to connect to our secret service, or generate the certificates and create the secrets manually.

This gets really tricky when you are doing docker in docker because the host path of the host doesn't match up properly with the host of the local machine. This also means that we can remove the "run as the current user id" hack to make sure that mounted files were accessible by the hosted user.

Crazybus · 2019-07-09T13:49:42Z

@mounaBoo Can you add some more information for me? I can't really do a whole lot with a "this failed" log message.

Some information that would help me:

The full output from the make command (not just the final line saying it failed)
Which Operating System/shell you are using
Output from these commands

helm get helm-es-security
helm get helm-kibana-security
kubectl get pods
kubectl get events
kubectl get secrets # also check that these secrets seem to have the correct output
kubectl logs -l app=security-master
kubectl logs -l release=helm-kibana-security

mounaBoo · 2019-07-09T14:23:03Z


kubectl delete secrets elastic-credentials elastic-certificates elastic-certificate-pem || true && \
vault read -field=value secret/devops-ci/helm-charts/elasticsearch/security/certificates | base64 --decode > elastic-certificates.p12 && \
vault read -field=value secret/devops-ci/helm-charts/elasticsearch/security/certificate-pem | base64 --decode > elastic-certificate.pem && \
kubectl create secret generic elastic-credentials  --from-literal=password=changeme --from-literal=username=elastic && \
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12 && \
kubectl create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem && \
rm -f elastic-certificates.p12 elastic-certificate.pem
secret "elastic-credentials" deleted
secret "elastic-certificates" deleted
secret "elastic-certificate-pem" deleted
Error reading secret/devops-ci/helm-charts/elasticsearch/security/certificates: Get https://127.0.0.1:8200/v1/secret/devops-ci/helm-charts/elasticsearch/security/certificates: dial tcp 127.0.0.1:8200: connect: connection refused
Error reading secret/devops-ci/helm-charts/elasticsearch/security/certificate-pem: Get https://127.0.0.1:8200/v1/secret/devops-ci/helm-charts/elasticsearch/security/certificate-pem: dial tcp 127.0.0.1:8200: connect: connection refused
secret/elastic-credentials created
secret/elastic-certificates created
secret/elastic-certificate-pem created
helm upgrade --wait --timeout=600 --install --values ./security.yml helm-es-security ../../ ; \

UPGRADE FAILED
Error: "helm-es-security" has no deployed releases
Error: UPGRADE FAILED: "helm-es-security" has no deployed releases
Makefile:8: recipe for target 'install' failed
make:  [install] Error 1

I am using Kubernetes goversion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

helm get helm-kibana-security :

REVISION: 1
RELEASED: Tue Jul  9 02:47:06 2019
CHART: kibana-7.2.0
USER-SUPPLIED VALUES:
elasticsearchHosts: https://security-master:9200
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: something_at_least_32_characters
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
path: /usr/share/kibana/config/certs
protocol: https
secretMounts:
- name: elastic-certificate-pem
  secretName: elastic-certificate-pem

COMPUTED VALUES:
affinity: {}
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
elasticsearchHosts: https://security-master:9200
elasticsearchURL: ""
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
fullnameOverride: ""
healthCheckPath: /app/kibana
httpPort: 5601
image: docker.elastic.co/kibana/kibana
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.2.0
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: something_at_least_32_characters
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
maxUnavailable: 1
nameOverride: ""
nodeSelector: {}
path: /usr/share/kibana/config/certs
podSecurityContext:
  fsGroup: 1000
priorityClassName: ""
protocol: https
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 500m
secretMounts:
- name: elastic-certificate-pem
  secretName: elastic-certificate-pem
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
serverHost: 0.0.0.0
service:
  annotations: {}
  nodePort: null
  port: 5601
  type: ClusterIP
serviceAccount: ""
tolerations: []
updateStrategy:
  type: Recreate

HOOKS:
MANIFEST:

---
# Source: kibana/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: helm-kibana-security-kibana-config
  labels:
    app: kibana
    release: "helm-kibana-security"
data:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: something_at_least_32_characters
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
---
# Source: kibana/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: helm-kibana-security-kibana
  labels:
    app: kibana
    release: "helm-kibana-security"
    heritage: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      name: http
      targetPort: 5601
  selector:
    app: kibana
    release: "helm-kibana-security"
---
# Source: kibana/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helm-kibana-security-kibana
  labels:
    app: kibana
    release: "helm-kibana-security"
spec:
  replicas: 1
  strategy:
    type: Recreate

  selector:
    matchLabels:
      app: kibana
      release: "helm-kibana-security"
  template:
    metadata:
      labels:
        app: kibana
        release: "helm-kibana-security"
      annotations:

        configchecksum: 8283046c51cd1f0cec8ba23f5f3a24cc7760352187eb24837a77a19aa38aad4
    spec:
      securityContext:
        fsGroup: 1000

      volumes:
        - name: elastic-certificate-pem
          secret:
            secretName: elastic-certificate-pem
        - name: kibanaconfig
          configMap:
            name: helm-kibana-security-kibana-config
      containers:
      - name: kibana
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000

        image: "docker.elastic.co/kibana/kibana:7.2.0"
        env:
          - name: ELASTICSEARCH_HOSTS
            value: "https://security-master:9200"
          - name: SERVER_HOST
            value: "0.0.0.0"
          - name: ELASTICSEARCH_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials

        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                http () {
                    local path="${1}"
                    set -- -XGET -s --fail

                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    fi

                    curl -k "$@" "https://localhost:5601${path}"
                }

                http "/app/kibana"
        ports:
        - containerPort: 5601
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500m

        volumeMounts:
          - name: elastic-certificate-pem
            mountPath:
          - name: kibanaconfig
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml

mounaBoo · 2019-07-09T14:31:39Z

helm get helm-es-security :

REVISION: 1
RELEASED: Tue Jul  9 02:30:16 2019
CHART: elasticsearch-7.2.0
USER-SUPPLIED VALUES:
clusterName: security
esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
nodeGroup: master
protocol: https
roles:
  data: "true"
  ingest: "true"
  master: "true"
secretMounts:
- name: elastic-certificates
  path: /usr/share/elasticsearch/config/certs
  secretName: elastic-certificates

COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: security
esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
esJavaOpts: -Xmx1g -Xms1g
esMajorVersion: 7
extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
fsGroup: ""
fullnameOverride: ""
httpPort: 9200
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.2.0
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources: {}
masterService: ""
masterTerminationFix: false
maxUnavailable: 1
minimumMasterNodes: 2
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity: {}
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
priorityClassName: ""
protocol: https
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 3
resources:
  limits:
    cpu: 1000m
    memory: 2Gi
  requests:
    cpu: 100m
    memory: 2Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
schedulerName: ""
secretMounts:
- name: elastic-certificates
  path: /usr/share/elasticsearch/config/certs
  secretName: elastic-certificates
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service:
  annotations: {}
  nodePort: null
  type: ClusterIP
sidecarResources: {}
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

HOOKS:
---
# helm-es-security-trbls-test
apiVersion: v1
kind: Pod
metadata:
  name: "helm-es-security-trbls-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "helm-es-security-cyvse-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.2.0"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'security-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "security-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "security-master"
---
# Source: elasticsearch/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: security-master-config
  labels:
    heritage: "Tiller"
    release: "helm-es-security"
    chart: "elasticsearch-7.2.0"
    app: "security-master"
data:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: security-master-headless
  labels:
    heritage: "Tiller"
    release: "helm-es-security"
    chart: "elasticsearch-7.2.0"
    app: "security-master"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  # Create endpoints also if the related pod isn't ready
  publishNotReadyAddresses: true
  selector:
    app: "security-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: security-master
  labels:
    heritage: "Tiller"
    release: "helm-es-security"
    chart: "elasticsearch-7.2.0"
    app: "security-master"
  annotations:
    {}

spec:
  type: ClusterIP
  selector:
    heritage: "Tiller"
    release: "helm-es-security"
    chart: "elasticsearch-7.2.0"
    app: "security-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: security-master
  labels:
    heritage: "Tiller"
    release: "helm-es-security"
    chart: "elasticsearch-7.2.0"
    app: "security-master"
spec:
  serviceName: security-master-headless
  selector:
    matchLabels:
      app: "security-master"
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: security-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi

  template:
    metadata:
      name: "security-master"
      labels:
        heritage: "Tiller"
        release: "helm-es-security"
        chart: "elasticsearch-7.2.0"
        app: "security-master"
      annotations:

        configchecksum: 649601f6dca600a1af30e62d1dd790243da5de0431bbd18c579a9dc70df90d2
    spec:
      securityContext:
        fsGroup: 1000

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "security-master"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates
        - name: esconfig
          configMap:
            name: security-master-config
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.2.0"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          {}

      containers:
      - name: "elasticsearch"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000

        image: "docker.elastic.co/elasticsearch/elasticsearch:7.2.0"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} https://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 2Gi

        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "security-master-0,security-master-1,security-master-2,"
          - name: discovery.seed_hosts
            value: "security-master-headless"
          - name: cluster.name
            value: "security"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials
          - name: ELASTIC_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials
        volumeMounts:
          - name: "security-master"
            mountPath: /usr/share/elasticsearch/data
          - name: elastic-certificates
            mountPath: /usr/share/elasticsearch/config/certs
          - name: esconfig
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml

mounaBoo · 2019-07-09T14:32:45Z

kubctl get pods :

NAME                                            READY   STATUS             RESTARTS   AGE
elasticsearch-master-0                          1/1     Running            0          17m
elasticsearch-master-1                          1/1     Running            0          7h39m
elasticsearch-master-2                          1/1     Running            0          13h
kibana-kibana-7bcfdc49c6-r6zlp                  1/1     Running            0          21m
security-master-0                               0/1     CrashLoopBackOff   7          17m
security-master-1                               0/1     CrashLoopBackOff   148        13h
security-master-2                               0/1     CrashLoopBackOff   88         7h39m

mounaBoo · 2019-07-09T14:37:31Z

kubectl get secrets # also check that these secrets seem to have the correct output
kubectl logs -l app=security-master

elastic-certificate-pem     Opaque                                1      31m
elastic-certificates        Opaque                                1      31m
elastic-credentials         Opaque                                2      31m

"Caused by: java.io.IOException: Short read of DER length",
"at sun.security.util.DerInputStream.getLength(DerInputStream.java:588) ~[?:?]",
"at sun.security.util.DerValue.init(DerValue.java:390) ~[?:?]",
"at sun.security.util.DerValue.<init>(DerValue.java:331) ~[?:?]",
"at sun.security.util.DerValue.<init>(DerValue.java:344) ~[?:?]",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1980) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1472) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:89) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]",
"at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]",
"at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]",
"at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]",
"at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:143) ~[?:?]",
"at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]",
"at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]",
"at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]",
"at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]",
"at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]",
"at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",

kubectl logs -l release=helm-kibana-security

no logs appear

Crazybus · 2019-07-09T15:29:30Z

This line tells me that you aren't running the version from this branch. These lines are now removed and the certificates are generated automatically.

Error reading secret/devops-ci/helm-charts/elasticsearch/security/certificates: Get https://127.0.0.1:8200/v1/secret/devops-ci/helm-charts/elasticsearch/security/certificates: dial tcp 127.0.0.1:8200: connect: connection refused

Can you make sure you are running this from the alright_keep_your_secrets branch.

mounaBoo · 2019-07-09T16:38:01Z

OK I will do it right now , I will clean up what I have and then report you back

mounaBoo · 2019-07-10T00:32:49Z

I installed from the banche you proposed,

when I run the test with make goss I ama getting this errors :

https://localhost:5601/app/kibana: status: Error: Get https://localhost:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
https://helm-kibana-security-kibana:5601/app/kibana: status: Error: Get https://helm-kibana-security-kibana:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting hea
ders)
Failures/Skipped:
https://localhost:5601/app/kibana: status: Error: Get https://localhost:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
https://helm-kibana-security-kibana:5601/app/kibana: status: Error: Get https://helm-kibana-security-kibana:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting hea
ders)
Total Duration: 2.001s
Count: 4, Failed: 2, Skipped: 0
Retrying in 5s (elapsed/timeout time: 2.002s/5m0s)

I exposed the deployement for test and to visualize the dashboard so I can see the login into the UI enabled but then it could connect.

this the message I am getting in my navigator :

The connection was reset

The connection to the server was reset while the page was loading.

The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

the whole output of make goss :

GOSS_CONTAINER=$(kubectl get --no-headers=true pods -l release=helm-kibana-security -o custom-columns=:metadata.name | sed -n 1p ) && \
echo Testing with pod: $GOSS_CONTAINER && \
kubectl cp test/goss.yaml $GOSS_CONTAINER:/tmp/goss.yaml && \
kubectl exec $GOSS_CONTAINER -- sh -c "cd /tmp/ && curl -s -L https://github.com/aelsabbahy/goss/releases/download/v0.3.6/goss-linux-amd64 -o goss && chmod +rx ./goss && ./goss --gossfile goss.yaml validate --retry-timeout 300s --sleep 5s --color --format documentation"
Testing with pod: helm-kibana-security-kibana-778dfcf8f9-p29zh
File: /usr/share/kibana/config/kibana.yml: exists: matches expectation: [true]
File: /usr/share/kibana/config/kibana.yml: contains: matches expectation: [server.ssl:   enabled: true   key: /usr/share/kibana/config/certs/elastic-certificate.pem   certificate: /usr/share/kiba
na/config/certs/elastic-certificate.pem xpack.security.encryptionKey: elasticsearch.ssl:   certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem   verificationMode: certi
ficate]
https://localhost:5601/app/kibana: status: Error: Get https://localhost:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
https://helm-kibana-security-kibana:5601/app/kibana: status: Error: Get https://helm-kibana-security-kibana:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting hea
ders)
Failures/Skipped:
https://localhost:5601/app/kibana: status: Error: Get https://localhost:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
https://helm-kibana-security-kibana:5601/app/kibana: status: Error: Get https://helm-kibana-security-kibana:5601/app/kibana: net/http: request canceled (Client.Timeout exceeded while awaiting hea
ders)
Total Duration: 2.001s
Count: 4, Failed: 2, Skipped: 0
Retrying in 5s (elapsed/timeout time: 2.002s/5m0s)
Attempt #2:
File: /usr/share/kibana/config/kibana.yml: exists: matches expectation: [true]
File: /usr/share/kibana/config/kibana.yml: contains: matches expectation: [server.ssl:   enabled: true   key: /usr/share/kibana/config/certs/elastic-certificate.pem   certificate: /usr/share/kiba
na/config/certs/elastic-certificate.pem xpack.security.encryptionKey: elasticsearch.ssl:   certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem   verificationMode: certi
ficate]
HTTP: https://helm-kibana-security-kibana:5601/app/kibana: status: matches expectation: [200]
HTTP: https://localhost:5601/app/kibana: status: matches expectation: [200]
Total Duration: 0.479s
Count: 4, Failed: 0, Skipped: 0

Crazybus · 2019-07-10T14:21:13Z

Attempt #2:
File: /usr/share/kibana/config/kibana.yml: exists: matches expectation: [true]
File: /usr/share/kibana/config/kibana.yml: contains: matches expectation: [server.ssl:   enabled: true   key: /usr/share/kibana/config/certs/elastic-certificate.pem   certificate: /usr/share/kiba
na/config/certs/elastic-certificate.pem xpack.security.encryptionKey: elasticsearch.ssl:   certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem   verificationMode: certi
ficate]
HTTP: https://helm-kibana-security-kibana:5601/app/kibana: status: matches expectation: [200]
HTTP: https://localhost:5601/app/kibana: status: matches expectation: [200]
Total Duration: 0.479s
Count: 4, Failed: 0, Skipped: 0

The last part of the logs show that this actually worked! The errors that you are seeing are because goss does retries on failure. Thank you for taking the time to test that this is actually working for someone besides me and the CI system.

From a bootstrapping point of view it looks like it is working for you. My suspicion is that you might have some other issues (e.g. with slow storage) that is causing things to be flaky. Could you open a separate issue with more detail so that we can investigate that further?

mounaBoo · 2019-07-10T14:57:26Z

ok I opened a new issue with more information about the error I am getting https://localhost:5601/app/kibana: status: Error #219

mounaBoo · 2019-07-11T11:11:48Z

kibana/README.md

  # Get the auto generated password
-  kubectl get secret elastic-credentials -o jsonpath='{.data.password}' | base64 --decode
-  kubectl port-forward deployment/helm-kibana-default-kibana 5601


kubectl port-forward deployment/helm-kibana-default-kibana 5601 sir I have a question about that don't you mean by that :

kubectl port-forward deployment/helm-kibana**-security-**kibana 5601

Yes, that is exactly what is in the code. I think you are looking at the part of the diff that was deleted rather than the new line which was added below.

that's ok it wasn't updated at the time I looked at it. but still having problem with kibana UI

tylerjl

Ran through the new security make targets locally (with some tweaks for minikube) and works for me 👍 I like auto-generating most of the material and just exec-ing into the pods to issue example requests with the secret values.

Crazybus requested a review from tylerjl July 5, 2019 10:26

[kibana] Fixup security install docs

dbe09b1

Fixes: #193 The default makefile target was switched at some point in time to also include installing secrets from our secret service. This obviously doesn't work so good for users.

Crazybus force-pushed the alright_keep_your_secrets branch from a31fc98 to 6c53336 Compare July 8, 2019 13:23

Update Kibana and Elasticsearch security examples to be generated

662171e

Closes: #27 This makes it possible to run all of the examples without needing to connect to our secret service, or generate the certificates and create the secrets manually.

Crazybus force-pushed the alright_keep_your_secrets branch from 6c53336 to 662171e Compare July 8, 2019 13:24

Mount docker socket into testing container

dd84f33

Crazybus removed the request for review from tylerjl July 8, 2019 14:24

Crazybus force-pushed the alright_keep_your_secrets branch from 0f2a564 to cb718a9 Compare July 8, 2019 18:15

Crazybus changed the title ~~[kibana] Fixup security install docs~~ Fixup security install docs Jul 8, 2019

Use generated encryption key for Kibana

d004692

Crazybus requested a review from tylerjl July 8, 2019 19:59

This was referenced Jul 9, 2019

Install Kibana + security failing #193

Closed

How to create security certificate pem for elasticsearch an kibana #211

Closed

UPGRADE FAILED: timed out waiting for the condition #209

Closed

Security enabling: Error: no objects visited #201

Closed

Fixup the instructions for how to actual interact with the deployment

1a10ea1

mounaBoo reviewed Jul 11, 2019

View reviewed changes

tylerjl approved these changes Jul 15, 2019

View reviewed changes

Crazybus merged commit 581b083 into master Jul 17, 2019

Crazybus deleted the alright_keep_your_secrets branch July 17, 2019 08:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixup security install docs #205

Fixup security install docs #205

Crazybus commented Jul 5, 2019 •

edited

Loading

Crazybus commented Jul 8, 2019

Crazybus commented Jul 9, 2019

mounaBoo commented Jul 9, 2019 •

edited

Loading

mounaBoo commented Jul 9, 2019

mounaBoo commented Jul 9, 2019 •

edited

Loading

mounaBoo commented Jul 9, 2019

Crazybus commented Jul 9, 2019

mounaBoo commented Jul 9, 2019

mounaBoo commented Jul 10, 2019 •

edited

Loading

Crazybus commented Jul 10, 2019

mounaBoo commented Jul 10, 2019

mounaBoo Jul 11, 2019

Crazybus Jul 12, 2019

mounaBoo Jul 12, 2019

tylerjl left a comment

Fixup security install docs #205

Fixup security install docs #205

Conversation

Crazybus commented Jul 5, 2019 • edited Loading

Crazybus commented Jul 8, 2019

Crazybus commented Jul 9, 2019

mounaBoo commented Jul 9, 2019 • edited Loading

mounaBoo commented Jul 9, 2019

mounaBoo commented Jul 9, 2019 • edited Loading

mounaBoo commented Jul 9, 2019

Crazybus commented Jul 9, 2019

mounaBoo commented Jul 9, 2019

mounaBoo commented Jul 10, 2019 • edited Loading

Crazybus commented Jul 10, 2019

mounaBoo commented Jul 10, 2019

mounaBoo Jul 11, 2019

Choose a reason for hiding this comment

Crazybus Jul 12, 2019

Choose a reason for hiding this comment

mounaBoo Jul 12, 2019

Choose a reason for hiding this comment

tylerjl left a comment

Choose a reason for hiding this comment

Crazybus commented Jul 5, 2019 •

edited

Loading

mounaBoo commented Jul 9, 2019 •

edited

Loading

mounaBoo commented Jul 9, 2019 •

edited

Loading

mounaBoo commented Jul 10, 2019 •

edited

Loading