NGINX not reloading correctly

[rikatz]

I'm using the latest Ingress Controller (following the 'master' branch), but using my on prod Kubernetes Cluster.

I have some Ingress misconfigured, and this is causing a loop message with them (ssl certs, etc), and also the continuous reload of the NGINX.

What happens here is that even with those reloads, it seems NGINX is not loading the nginx.conf file correctly.

The following is what happens:

• NGINX starts with the empty/start nginx.conf file
• Ingress Controller inserts the default_backend conf, and reloads nginx correctly
• Ingress Controller tries to fetch the remaining config, does this correctly (the final nginx.conf contains all the vHosts) but is not reloading NGINX (even it prints the message about the reload).

The updates just seems to work, after all, when you change any Ingress Object. After changing any object, Nginx reloads correctly.

Thanks

[aledbf]

@rikatz please test the image quay.io/aledbf/nginx-ingress-controller:0.185

[aledbf]

@rikatz please post the logs of the ingress pod

[rikatz]

@aledbf Your image is using the bogused nginx.tmpl (https://github.com/kubernetes/ingress/blob/master/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl#L576)

I'm replacing it to $server.Hostname and will post the logs.

[rikatz]

@aledbf I have some confidential data here, but will try to clean the logs.

Here's what happens:

W0821 19:08:41.685915       5 controller.go:956] service gscvi-t/appfla does not have any active endpoints
W0821 19:08:41.692205       5 controller.go:887] error obtaining service endpoints: service ouvserpro-d/dummy-soto-jboss does not exist
W0821 19:08:41.696117       5 controller.go:956] service namespace12/api-siafi does not have any active endpoints
W0821 19:08:41.702834       5 controller.go:956] service gscvi-p/ws-custos does not have any active endpoints
W0821 19:08:41.705346       5 controller.go:956] service gscvi-t/cerebro does not have any active endpoints
W0821 19:08:41.727107       5 controller.go:956] service namespace110-d/backend does not have any active endpoints
W0821 19:08:41.732101       5 controller.go:956] service namespace105-h/nginx-estatico does not have any active endpoints
W0821 19:08:41.736048       5 controller.go:887] error obtaining service endpoints: service namespace11/nginx-estatico does not exist
W0821 19:08:41.743510       5 controller.go:956] service namespace101-p/backend does not have any active endpoints
W0821 19:08:41.746072       5 controller.go:956] service namespace101-p/tesseract does not have any active endpoints
W0821 19:08:41.751963       5 controller.go:887] error obtaining service endpoints: service gsopi-p/httpd-gsopi does not exist
W0821 19:08:41.782882       5 controller.go:887] error obtaining service endpoints: service namespace100-h/jboss-events does not exist
W0821 19:08:41.795289       5 controller.go:887] error obtaining service endpoints: service namespace11/teste does not exist
W0821 19:08:41.799641       5 controller.go:887] error obtaining service endpoints: service namespace11/cors does not exist
W0821 19:08:41.810235       5 controller.go:956] service namespace100-p/jboss-auth-preprod does not have any active endpoints
W0821 19:08:41.810809       5 controller.go:956] service namespace100-p/jboss-servicos-preprod does not have any active endpoints
W0821 19:08:41.810924       5 controller.go:887] error obtaining service endpoints: service namespace100-p/jboss-authn-preprod does not exist
W0821 19:08:41.820350       5 controller.go:887] error obtaining service endpoints: service namespace15/backend-core does not exist
W0821 19:08:41.848247       5 controller.go:956] service namespace9-v/service-jboss does not have any active endpoints
I0821 19:08:41.881847       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.881958       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.881997       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882117       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882204       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.882328       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882422       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882502       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882618       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.882833       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883029       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883082       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883339       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883440       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883523       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883656       5 controller.go:1137] ssl certificate "namespace1/secret1" does not exist in local store
I0821 19:08:41.883709       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.883791       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.883903       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.884172       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.884253       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.884366       5 controller.go:1137] ssl certificate "namespace8/secret3" does not exist in local store
I0821 19:08:41.884453       5 controller.go:1137] ssl certificate "namespace8/secret3" does not exist in local store
I0821 19:08:41.884499       5 controller.go:1137] ssl certificate "namespace8/secret3" does not exist in local store
I0821 19:08:41.884645       5 controller.go:1137] ssl certificate "namespace8/secret3" does not exist in local store
I0821 19:08:41.884851       5 controller.go:1137] ssl certificate "namespace3/secret2" does not exist in local store
I0821 19:08:41.885053       5 controller.go:1137] ssl certificate "namespace4/secret2" does not exist in local store
I0821 19:08:41.885145       5 controller.go:1137] ssl certificate "namespace4/secret2" does not exist in local store
I0821 19:08:41.885189       5 controller.go:1137] ssl certificate "namespace4/secret2" does not exist in local store
I0821 19:08:41.885284       5 controller.go:1137] ssl certificate "namespace4/secret2" does not exist in local store
I0821 19:08:41.885396       5 controller.go:1137] ssl certificate "namespace4/secret2" does not exist in local store
I0821 19:08:41.885561       5 controller.go:1137] ssl certificate "namespace120-v/secret-tls" does not exist in local store
I0821 19:08:41.886028       5 controller.go:1137] ssl certificate "namespace5/secret-tls" does not exist in local store
I0821 19:08:41.886066       5 controller.go:1137] ssl certificate "namespace5/secret-tls" does not exist in local store
I0821 19:08:41.886164       5 controller.go:1137] ssl certificate "namespace6/secret-tls" does not exist in local store
I0821 19:08:41.886264       5 controller.go:1137] ssl certificate "namespace6/secret-tls" does not exist in local store
I0821 19:08:41.886367       5 controller.go:1137] ssl certificate "namespace7/secret-tls" does not exist in local store
I0821 19:08:41.886459       5 controller.go:1137] ssl certificate "namespace7/secret-tls" does not exist in local store
I0821 19:08:41.886647       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:41.886696       5 controller.go:1137] ssl certificate "namespace2/secret2" does not exist in local store
I0821 19:08:42.396432       5 controller.go:423] backend reload required
I0821 19:08:43.110050       5 controller.go:432] ingress backend successfully reloaded...

This is in a loop, and while nginx.conf is configured correctly, nginx is not reloading.

After changing the ingress object of one of the cases (like namespace2/ingress), ingress reloads correctly NGINX

[rikatz]

BTW, I'm using your image:

I0821 19:08:34.930954       5 launch.go:108] &{NGINX 0.9.0-beta.11 git-827d8520 https://github.com/aledbf/ingress}
I0821 19:08:34.931304       5 launch.go:111] Watching for ingress class: nginx
I0821 19:08:34.933678       5 launch.go:270] Creating API server client for https://master.kubernetes
I0821 19:08:34.940218       5 nginx.go:168] starting NGINX process...
I0821 19:08:35.015667       5 launch.go:127] validated estaleiro/ingress-default-backend as the default backend
W0821 19:08:35.016735       5 controller.go:311] Update of ingress status is disabled (flag --update-status=false was specified)
I0821 19:08:35.017134       5 controller.go:1280] starting Ingress controller

[aledbf]

@rikatz this works as expected. Please keep in mind that any update (or initial sync) to ingress, secrets, endpoints and configmaps are asynchronous.

Edit: the real issue here is the duplication of the warning in the log.

[rikatz]

@aledbf but Ingress only starts answering correctly after someone changes any Ingress Object.

Before this, NGINX directs all the ingresses to default backend.

[aledbf]

@rikatz please update the image to quay.io/aledbf/nginx-ingress-controller:0.186

[rikatz]

OK, the template is ok, but the error still happens.

I'm looking here to see if there's some secret causing this, or maybe this initial (a)sync is causing some issue to Ingress Controller.

[rikatz]

@aledbf So it seems it's working, but Ingress is taking too much time to load each of the secrets.

The CA certificates are loaded immediately, but the TLS certs are not being loaded as fast as the CA. Isn't the case to load all the available secrets before reloading, in initial sync? And then reload as necessary?

Thanks!

[aledbf]

@rikatz please test quay.io/aledbf/nginx-ingress-controller:0.187. This image includes an initial sync of the secrets.

[rikatz]

Yeap, this is working as expected o/