Distribute BE instances (pods) evenly across available nodes

[rameshpolishetti]

Context
Enable Kubernetes native pod affinity functionality through BE helm chart, so that the BE instances (pods) distribute evenly across available nodes (physical boxes).

Alternatives
No

Has the feature been requested before?
No

If the feature request is approved, would you be willing to submit a PR?
Yes

[LakshmiMekala]

Required implementation is completed and available with PR #194

[rameshpolishetti]

This is made available in PR #213 since previous PR got dropped.

[tsaxena18]

Tried with setting it to true and my EKS cluster has 4 nodes. As per definition of podAntiAffinity, no two pods can be on same node as per defined rule. Here it does not seem to be working for me as per my understanding. I will continue to debug more into it.
[apps@tsaxena-z820 kubernetes]$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ccapp-cacheagent-0 1/1 Running 0 4m35s 192.168.44.232 ip-192-168-33-68.us-east-2.compute.internal
ccapp-cacheagent-1 1/1 Running 0 4m34s 192.168.26.82 ip-192-168-1-38.us-east-2.compute.internal
ccapp-cacheagent-2 1/1 Running 0 4m31s 192.168.71.5 ip-192-168-85-167.us-east-2.compute.internal
ccapp-cacheagent-3 1/1 Running 0 4m28s 192.168.41.254 ip-192-168-35-250.us-east-2.compute.internal
ccapp-cacheagent-4 1/1 Running 0 4m26s 192.168.3.182 ip-192-168-1-38.us-east-2.compute.internal
ccapp-cacheagent-5 1/1 Running 0 4m24s 192.168.74.112 ip-192-168-85-167.us-east-2.compute.internal
ccapp-inferenceagent-0 1/1 Running 0 4m35s 192.168.78.132 ip-192-168-85-167.us-east-2.compute.internal

[LakshmiMekala]

@tsaxena18 this feature defines pods are evenly distributed across nodes. The above mentioned output of nodes is correct

[tsaxena18]

There are two types of node affinity, called requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution and they are like hard/soft rule as mentioned in https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/.

Currently, I can see the latter, preferredDuringSchedulingIgnoredDuringExecution, is defined in our helm script and if I change it to other one, then install throws error so I assume we have to stick with this option only.
We can close this ticket as the behavior is expected as per rule defined in script.

[apps@tsaxena-z820 kubernetes]$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ccapp-cacheagent-0 1/1 Running 0 12s 192.168.73.171 ip-192-168-75-15.us-east-2.compute.internal
ccapp-cacheagent-1 1/1 Running 0 10s 192.168.52.142 ip-192-168-41-21.us-east-2.compute.internal
ccapp-cacheagent-2 1/1 Running 0 9s 192.168.10.255 ip-192-168-27-216.us-east-2.compute.internal
ccapp-cacheagent-3 1/1 Running 0 7s 192.168.71.10 ip-192-168-85-244.us-east-2.compute.internal
ccapp-inferenceagent-0 1/1 Running 0 12s 192.168.56.99 ip-192-168-41-21.us-east-2.compute.internal
ccapp-inferenceagent-1 1/1 Running 0 11s 192.168.18.113 ip-192-168-27-216.us-east-2.compute.internal
ccapp-inferenceagent-2 1/1 Running 0 9s 192.168.86.98 ip-192-168-85-244.us-east-2.compute.internal

Let me know if we need to create a ticket to support other type, i.e, requiredDuringSchedulingIgnoredDuringExecution.

[LakshmiMekala]

@tsaxena18 we are aware of this features wrt podAntiAffinity, we have requirement to distribute pods across nodes evenly. U can close this issue. I dont think it is required to create a new support ticket for other type

[LakshmiMekala]

Closing as per @tsaxena18 comments