Scaling/design of default-http-backend Deployment

[dkeightley]

What kind of request is this (question/bug/enhancement/feature request): enhancement

Steps to reproduce (least amount of steps as possible): In environments where a high volume of requests to ingress-nginx are performed where there is no matching Ingress rule for the requests, increased load is placed on the default-http-backend Pod to handle these requests. Metric collection could also be influencing resource usage during this load.

Result: The default default-http-backend deployment configuration (1 replica, and CPU/memory resource limits) can introduce a performance and stability issue, with bursty response times and OOM kill of the container.

Maintaining this configuration would benefit from scaling the default-http-backend Deployment by default, with a minimum of 2 replicas and anti-affinity. The default resource limits may need reviewing to apply to a wider use case.

Other details that may be helpful:

As ingress-nginx and the default-http-backend share areas of concern, and mutual dependencies, it may make sense to tightly couple these containers in the ingress-nginx DaemonSet.

edit: The /healthz endpoint traffic measured was from the kubelet readiness/liveness probes - not external healthcheck requests, ingress-nginx handles these locally, previous mention of /healthz and healthchecks can be ignored.

gz#10803

gz#10926

gz#11108

[spencergilbert]

based on kubernetes/ingress-nginx#3196 it looks like you could also rely on the nginx pods themselves

[maggieliu]

Details on what will be implemented:

Provide a configuration method for RKE provisioned clusters where the default backend service isn't passed.

ingress:
  provider: nginx
     default-backend: false

Should remove the --default-backend-service flag from the nginx ingress controller add on and not deploy the default-http-backend service/pod.

This is a breaking change in behavior and should be an option. When the next version of Kubernetes ships we should make it the default.

This should take the same approach as upstream: kubernetes/ingress-nginx#3196

This is also captured in #25590

[cbron]

@nickgerace once this is reviewed/merged can you make a docs issue and PR for this.

[nickgerace]

@cbron not a problem

[bmdepesa]

rancher/rancher:v2.5.4-rc3
rancher/rancher:master-head 46b0df4

• Create a cluster
  • Under advanced options, Nginx Default Backend is disabled
  • Edit yaml, ensure nginx.ingress.default_backend is false
  • Create cluster
• In the System project's ingress-nginx namespace, default-http-backend is not deployed
• Edit the cluster, setting the value to enabled
• Cluster should update and the default-http-backend workload should be deployed

---

• Create a cluster
  • Under advanced options, Nginx Default Backend is disabled
  • Set to enabled
  • Edit yaml, ensure nginx.ingress.default_backend is true
  • Create cluster
• In the System project's ingress-nginx namespace, default-http-backend is deployed
• Edit the cluster, setting the value to disabled
• Cluster should update and the default-http-backend workload should be removed

---

• Install Rancher v2.5.3
• Create a cluster
• default-http-backend should be deployed
• Upgrade to v2.5.4-rc3
• Edit cluster
  • The setting should be set to enabled to reflect the current state
  • Set to disabled
• The cluster should update and the default-http-backend workload should be removed

@nickgerace:
Before closing, we just want to confirm that the UI's default behavior should be to have the default backend disabled, while in RKE 1.2.4 the default behavior is to have it enabled.

[nickgerace]

They should both be disabled by default.

This is a breaking change in behavior and should be an option. When the next version of Kubernetes ships we should make it the default.
#25590 (comment)

We were originally going back and forth with the default behavior, but decided to land on maintaining RKE behavior. There's a ticket to track when Kubernetes 1.20 becomes the default version, and then we will "flip" the behavior in both the UI and API.
Reference: #30356