App creates constant CPU load

[d-uzlov]

Expected Behavior

When there are no new requests and no network changes CPU load should be close to zero.

Current Behavior

If you just deploy nse-vl3-vpp app you will get constant CPU load from this app and from forwarder.

Steps to Reproduce

1. Deploy NSM basic setup
2. Deploy metrics server
   • https://github.com/kubernetes-sigs/metrics-server
3. Do first few steps from vl3-basic example
   • You only need to create namespace and apply the kustomization file
   • You can edit the kustomization file and disable clients. When I tested this I got this bug even without any clients
4. run kubectl top pod --all-namespaces
   • nse-vl3-vpp and a forwarder will have high CPU load all the time, even though there are no requests and no network changes

Context

• kind version: 0.17.0
• kubectl client version: 1.25.2
• kustomize version: 4.5.7
• kubectl server version: 1.25.3
• Deployments ref: 9eeff3941ac2efd87d1e3775e7c4803474078192

Failure Logs

vl3-cluster-dump.zip

NAMESPACE            NAME                                           CPU(cores)   MEMORY(bytes)   
kube-system          coredns-565d847f94-5t9d7                       3m           13Mi            
kube-system          coredns-565d847f94-ppdn6                       2m           13Mi            
kube-system          etcd-kind-2-control-plane                      43m          31Mi            
kube-system          kindnet-wbnw6                                  1m           7Mi             
kube-system          kube-apiserver-kind-2-control-plane            47m          364Mi           
kube-system          kube-controller-manager-kind-2-control-plane   26m          42Mi            
kube-system          kube-proxy-tkpsc                               1m           10Mi            
kube-system          kube-scheduler-kind-2-control-plane            4m           17Mi            
kube-system          metrics-server-69dcc44d99-9r59n                10m          17Mi            
local-path-storage   local-path-provisioner-684f458cdd-srkqv        2m           6Mi             
metallb-system       controller-6658b8446c-2hfff                    2m           10Mi            
metallb-system       speaker-kssjt                                  5m           12Mi            
ns-vl3               nse-vl3-vpp-6c5b77956f-4trh7                   334m         203Mi           
ns-vl3               nse-vl3-vpp-6c5b77956f-gm7fn                   326m         203Mi           
ns-vl3               vl3-ipam-5cd4f7879c-9fdhs                      1m           4Mi             
nsm-system           admission-webhook-k8s-6664486479-2k8cd         1m           9Mi             
nsm-system           forwarder-vpp-vstbl                            331m         217Mi           
nsm-system           nsmgr-65msj                                    30m          25Mi            
nsm-system           nsmgr-proxy-697c66994d-j7jx4                   1m           11Mi            
nsm-system           registry-k8s-599f555cf-4x5jb                   3m           11Mi            
nsm-system           registry-proxy-7fbbf5cfd7-kqzfk                1m           4Mi             
spire                spire-agent-nsnt4                              43m          30Mi            
spire                spire-server-0                                 11m          37Mi             

[glazychev-art]

So, this is normal behavior and it is related to VPP. We use this framework for cmd-forwarder-vpp and for cmd-nse-vl3-vpp.
These applications create memif interfaces. In the case of cmd-nse-vl3-vpp, they are always created, even without external clients, because the NSE is a client to itself. Memifs work in polling mode:

vpp# sh int rx-placement
Thread 0 (vpp_main):
 node memif-input:
    memif0/0 queue 0 (polling)

As we can see from the documentation - With at least one interface in polling mode, the VPP CPU utilization is always 100% - https://fd.io/docs/vpp/v2101/troubleshooting/cpuusage.html#vpp-cpu-load

Since the limits for forwarder are quite high, that's why we see such a load.

[denis-tingaikin]

Seems like @glazychev-art is correct.

@edwarnicke Do you have any ideas on this?

[edwarnicke]

@glazychev-art I've looked at the code, and we seem to be trying to set memif interfaces to adaptive polling:

https://github.com/networkservicemesh/sdk-vpp/blob/ef0d5f51a8c659b8ff4fa4d8e7816666daef3a53/pkg/networkservice/mechanisms/memif/common.go#L155-L166

the question is: why isn't that happening?

[glazychev-art]

So, we solved the problem with memifs. It turned out that the adaptive mode was not set.

This greatly reduced the load on the CPU:

kube-system          coredns-565d847f94-kpzf8                     4m           14Mi            
kube-system          coredns-565d847f94-lnzgc                     3m           13Mi            
kube-system          etcd-kind-control-plane                      30m          35Mi            
kube-system          kindnet-bmk9k                                1m           8Mi             
kube-system          kindnet-l9bd4                                1m           8Mi             
kube-system          kindnet-zr74z                                1m           8Mi             
kube-system          kube-apiserver-kind-control-plane            53m          311Mi           
kube-system          kube-controller-manager-kind-control-plane   19m          47Mi            
kube-system          kube-proxy-nqsr4                             1m           11Mi            
kube-system          kube-proxy-qsn5g                             1m           10Mi            
kube-system          kube-proxy-s5t4z                             1m           10Mi            
kube-system          kube-scheduler-kind-control-plane            3m           19Mi            
kube-system          metrics-server-696f4844bb-2qfxw              5m           18Mi            
local-path-storage   local-path-provisioner-684f458cdd-8rrdx      1m           7Mi             
ns-vl3               alpine-79bf8df9bb-j9p72                      5m           10Mi            
ns-vl3               alpine-79bf8df9bb-lnvgv                      7m           10Mi            
ns-vl3               nse-vl3-vpp-6b6dcf5cfd-ms5s8                 21m          207Mi           
ns-vl3               nse-vl3-vpp-6b6dcf5cfd-xbkcc                 22m          208Mi           
ns-vl3               vl3-ipam-649866f85b-t64dm                    0m           5Mi             
nsm-system           admission-webhook-k8s-5c5cf9dc4c-jd5jw       1m           8Mi             
nsm-system           forwarder-vpp-4d6zp                          99m          223Mi           
nsm-system           forwarder-vpp-5q8b2                          115m         223Mi           
nsm-system           nsmgr-dvgfj                                  32m          29Mi            
nsm-system           nsmgr-fl5xl                                  21m          29Mi            
nsm-system           registry-k8s-7dcc4c7c95-wnvxg                5m           16Mi            
spire                spire-agent-pmx2x                            36m          42Mi            
spire                spire-agent-rbh4n                            33m          40Mi            
spire                spire-server-0                               12m          49Mi 

[denis-tingaikin]

Fixed since v1.7.0-rc.2