fix: allow redirection on the readinessProbe

[kuisathaverat]

What does this PR do?

It adds the -L flag to curl to allow redirections.

Why is it important?

It fixes the readinessProbe for kibana 8.0.0-SNAPSHOT version.

Related issues

Closes #548

[jmlrt]

⛴

[andrew-pickin-epi]

This change does not fix the issue.

    Readiness:  exec [sh -c #!/usr/bin/env bash -e
http () {
    local path="${1}"
    set -- -XGET -s --fail -L

    if [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; then
      set -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"
    fi

    STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601${path}")
    if [[ "${STATUS}" -eq 200 ]]; then
      exit 0
    fi

    echo "Error: Got HTTP code ${STATUS} but expected a 200"
    exit 1
}

http "/app/kibana"
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      ELASTICSEARCH_HOSTS:  http://elasticsearch-master:9200
      SERVER_HOST:          0.0.0.0
      NODE_OPTIONS:         --max-old-space-size=1800
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rlvcr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-rlvcr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rlvcr
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                                                   Message
  ----     ------     ----              ----                                                   -------
  Normal   Scheduled  <unknown>         default-scheduler                                      Successfully assigned logging/kibana-kibana-64d6c74457-wl5h5 to ip-192-168-58-183.eu-west-1.compute.internal
  Normal   Pulled     32s               kubelet, ip-192-168-58-183.eu-west-1.compute.internal  Container image "docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT" already present on machine
  Normal   Created    32s               kubelet, ip-192-168-58-183.eu-west-1.compute.internal  Created container kibana
  Normal   Started    32s               kubelet, ip-192-168-58-183.eu-west-1.compute.internal  Started container kibana
  Warning  Unhealthy  9s (x2 over 19s)  kubelet, ip-192-168-58-183.eu-west-1.compute.internal  Readiness probe failed: Error: Got HTTP code 000 but expected a 200

[kuisathaverat]

@andrew-pickin-epi I think your error is different, in your case the return code is 000 not 302 or 304, so in your case, there is no redirection there is a more serious error. the 000 code is returned by cURLwhen the command fails for some reason

https://superuser.com/questions/501690/curl-http-code-of-000

[andrew-pickin-epi]

Maybe so, but the point is that the readiness isn't succeeding when the system is working.

$ kubectl -n logging exec -it kibana-kibana-64d6c74457-pw2cw -- curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601/app/kibana"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  102k  100  102k    0     0  3051k      0 --:--:-- --:--:-- --:--:-- 3104k
200

Clearly this returns a 200 when run directly from an exec.

If I remove the output to /dev/null I see valid html. I'll not post this for brevity, but if you wish to see it, it's available.

[andrew-pickin-epi]

Ok I think I have an understanding of the what it happening here. The default initial delay of 10s is much too short.
If the readiness probe fire before the application is running curl fails and status is 0.

$ kubectl -n logging exec -it $(kubectl -n logging get pod kibana-kibana-64d6c74457-k642w -- curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601/app/kibana"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to ::1: Cannot assign requested address
000command terminated with exit code 7

I suggest a much bigger default initial delay or a startup probe or a readiness probe that takes account of curl failing before the network is established.

Note the is no echo for a successful readiness probe, this can result in the last event being recorded as unhealthy. This can be potentially confusing.