Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dns_lookup(): failed to query the DNS server: no AAAA record resolved, context: ngx.timer error #9248

Closed
marcandrews opened this issue Nov 2, 2022 · 7 comments
Labels
needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@marcandrews
Copy link

marcandrews commented Nov 2, 2022

What happened:

The nginx-ingress-controller appears to be functioning correctly, but is emitting these tens of thousands of these errors every hour:

2022/11/05 13:45:24 [error] 31#31: *49320 [lua] dns.lua:152: dns_lookup(): failed to query the DNS server for 10.16.0.23:
no AAAA record resolved, context: ngx.timer
server returned error code: 3: name error
2022/11/05 13:45:24 [error] 32#32: *49311 [lua] dns.lua:152: dns_lookup(): failed to query the DNS server for 10.16.4.24:
no AAAA record resolved, context: ngx.timer
server returned error code: 3: name error

What you expected to happen:

I expect no errors to be emitted.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
  Release:       v1.4.0
  Build:         50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:40:59Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12-gke.2300", GitCommit:"e55564cf3a1384026a54920174977659c8c56a50", GitTreeState:"clean", BuildDate:"2022-08-16T09:24:51Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: GKE

  • OS (e.g. from /etc/os-release): n/a

  • Kernel (e.g. uname -a): n/a

  • Install tools: Google Cloud Console

  • Basic cluster related info: n/a

  • How was the ingress-nginx-controller installed:

    helm upgrade --install ingress-nginx ingress-nginx \
      --repo https://kubernetes.github.io/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set rbac.create=true \
      --set controller.service.type=LoadBalancer \
      --set controller.service.externalTrafficPolicy=Local \
      --set controller.publishService.enabled=true
  • Current State of the controller:

    • kubectl describe ingressclasses
      Name:         nginx
      Labels:       app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.4.0
                    helm.sh/chart=ingress-nginx-4.3.0
      Annotations:  meta.helm.sh/release-name: ingress-nginx
                    meta.helm.sh/release-namespace: ingress-nginx
      Controller:   k8s.io/ingress-nginx
      Events:       <none>
      
    • kubectl -n <ingresscontrollernamespace> get all -A -o wide
      NAMESPACE        NAME                                                                 READY   STATUS      RESTARTS        AGE     IP            NODE                                                  NOMINATED NODE   READINESS GATES
      cert-manager     pod/cert-manager-7b8d75c477-55nn9                                    1/1     Running     2 (3d5h ago)    5d22h   10.16.4.2     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      cert-manager     pod/cert-manager-cainjector-6cd8d7f84b-swll8                         1/1     Running     10 (3d5h ago)   5d22h   10.16.0.2     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      cert-manager     pod/cert-manager-webhook-64d76db6c-t568k                             1/1     Running     0               5d21h   10.16.0.14    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      cert-manager     pod/cert-manager-webhook-64d76db6c-ttd66                             0/1     Completed   0               5d22h   10.16.5.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      custom-metrics   pod/custom-metrics-stackdriver-adapter-848d58c7c7-zjjwb              1/1     Running     0               5d22h   10.16.0.3     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/mongodb-0                                                        1/1     Running     0               5d22h   10.16.4.10    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/mongodb-1                                                        1/1     Running     0               5d22h   10.16.0.11    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/mongodb-2                                                        1/1     Running     0               5d22h   10.16.4.12    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-api-7b5c9dd98b-24nff                              1/1     Running     0               3d22h   10.16.4.24    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-api-7b5c9dd98b-2bdxj                              1/1     Running     0               3d22h   10.16.0.23    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-api-7b5c9dd98b-5gp2g                              1/1     Running     0               3d22h   10.16.0.24    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-api-7b5c9dd98b-tc5p2                              1/1     Running     1 (3d22h ago)   3d22h   10.16.4.23    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-25qv7                     1/1     Running     4 (5d22h ago)   5d22h   10.16.4.6     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-67s99                     1/1     Running     0               5d22h   10.16.0.8     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-extension-api-64964c8b7-bnqvr                     1/1     Running     4 (5d22h ago)   5d22h   10.16.4.5     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-vrfc7                     1/1     Running     0               5d22h   10.16.0.5     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5g54z                0/1     Error       0               5d5h    10.16.5.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5prjr                0/1     Error       0               3d21h   10.16.1.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5s4t6                0/1     Error       1 (3d23h ago)   3d23h   10.16.1.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-6sslf                0/1     Error       0               23h     10.16.1.17    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-77dtz                1/1     Running     0               20h     10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-7mqg4                0/1     Error       0               4d22h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-7tjfg                0/1     Error       0               45h     10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-857rm                1/1     Running     0               54m     10.16.1.19    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-8lcdn                0/1     Error       0               3d21h   10.16.1.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-8x49l                0/1     Error       0               2d22h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-9nn5c                0/1     Error       0               3d21h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bbsvn                0/1     Error       0               5d21h   10.16.5.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bpsmq                0/1     Error       0               3d21h   10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bz5k5                0/1     Error       0               45h     10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-c6rg2                0/1     Error       0               4d23h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-ccp9c                0/1     Error       0               47h     10.16.1.11    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-cqbsk                0/1     Error       0               5d      10.16.5.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dbs7b                0/1     Error       0               2d20h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dfh6c                0/1     Error       0               3d      10.16.1.16    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dj6vk                0/1     Error       0               2d22h   10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dtkm2                0/1     Error       0               3d21h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dv5kz                0/1     Error       0               5d21h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-fm9jk                0/1     Error       0               5d22h   10.16.5.11    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-g8gvm                1/1     Running     0               4h18m   10.16.1.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-gpv72                0/1     Error       0               3d21h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-gxl9c                0/1     Error       0               45h     10.16.1.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-hmcst                0/1     Error       0               45h     10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-hq7nt                0/1     Error       0               3d21h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-jqpqr                0/1     Error       0               5d21h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kfnpr                0/1     Error       0               4d22h   10.16.1.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kjbvc                0/1     Error       0               5d22h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kt2tv                0/1     Error       0               5d5h    10.16.5.14    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-lm46m                0/1     Error       0               2d22h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-lmrnn                0/1     Error       0               23h     10.16.1.18    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-mm6xt                0/1     Error       0               2d23h   10.16.1.17    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-nb8qj                0/1     Error       0               2d22h   10.16.1.18    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-ncpjh                0/1     Error       0               3d21h   10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-nfvpn                0/1     Error       0               3d21h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-q7m4q                0/1     Error       0               4d22h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-qxwb7                0/1     Error       0               5d22h   10.16.5.13    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rgjxc                1/1     Running     0               19h     10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rn89g                0/1     Error       0               4d22h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rrcws                0/1     Error       0               2d19h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rtlbd                1/1     Running     0               20h     10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-sgljb                0/1     Error       0               4d23h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-szjh7                0/1     Error       0               4d22h   10.16.5.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-t85jl                0/1     Error       0               3d21h   10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-v56rf                0/1     Error       0               5d22h   10.16.5.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-v7qzv                0/1     Error       0               4d22h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-vnrk2                0/1     Error       0               2d22h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wdh6j                0/1     Error       0               4d23h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wgzbw                0/1     Error       0               4d22h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wqcdc                0/1     Error       0               45h     10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-zlcgq                1/1     Running     0               39m     10.16.1.23    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-renderer-66959b6dc5-sqdmg                         1/1     Running     1 (5d22h ago)   5d22h   10.16.4.7     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-webhook-api-86bb7d96fd-6gxmv                      1/1     Running     0               5d22h   10.16.0.7     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-webhook-api-86bb7d96fd-kqgxv                      1/1     Running     4 (5d22h ago)   5d22h   10.16.4.8     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/redis-node-0                                                     2/2     Running     0               5d22h   10.16.0.13    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/redis-node-1                                                     2/2     Running     0               5d22h   10.16.4.11    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/redis-node-2                                                     2/2     Running     0               5d22h   10.16.4.13    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      ingress-nginx    pod/ingress-nginx-controller-69fbfb4bfd-k29tz                        1/1     Running     0               25m     10.16.1.25    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/event-exporter-gke-5479fd58c8-s44rs                              0/2     Completed   0               5d22h   10.16.5.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/event-exporter-gke-f66d9f855-8c7q2                               2/2     Running     0               2d22h   10.16.0.29    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/event-exporter-gke-f66d9f855-fqzvq                               0/2     Completed   0               3d5h    10.16.1.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/fluentbit-gke-dmnj9                                              2/2     Running     0               4d6h    10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/fluentbit-gke-kkmb2                                              2/2     Running     0               4d6h    10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/fluentbit-gke-qxwvq                                              2/2     Running     0               4d6h    10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/gke-metrics-agent-56dbf                                          1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/gke-metrics-agent-94xp8                                          1/1     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/gke-metrics-agent-ffz8h                                          1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-5hsqg                              1/1     Running     0               18h     10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-8vxw7                              0/1     Error       0               5d22h   10.16.5.2     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-dhk5j                              0/1     Error       0               5d21h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-hfc79                              0/1     Error       0               3d21h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-qk5p9                              0/1     Error       0               4d22h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-rnd6b                              0/1     Error       0               3d21h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-tfm5n                              1/1     Running     0               5d22h   10.16.0.12    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-wmjpj                              1/1     Running     0               5d21h   10.16.4.14    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-xp9jj                              0/1     Error       0               4d22h   10.16.5.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-xvm4m                              0/1     Error       0               45h     10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-z2nsx                              0/1     Error       0               2d14h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-autoscaler-6dfb4f9cfb-5cd56                   1/1     Running     0               5d21h   10.16.0.16    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-autoscaler-6dfb4f9cfb-mkl7h                   0/1     Error       0               5d22h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-autoscaler-f4d55555-677qc                               0/1     Error       0               5d22h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-autoscaler-f4d55555-npl4l                               1/1     Running     0               5d21h   10.16.0.15    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/kube-dns-b99c7865d-8bc5p                                         4/4     Running     0               5d22h   10.16.1.2     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-b99c7865d-rq5r7                                         4/4     Running     0               5d22h   10.16.0.10    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/kube-proxy-gke-****-preemptive-pool-e2-e219fbd6-54gl   1/1     Running     0               3d22h   10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-proxy-gke-****-standard-pool-e2-bae76619-3sf5     1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/kube-proxy-gke-****-standard-pool-e2-bae76619-ejn4     1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/metrics-server-v0.4.5-fb4c49dd6-tnp58                            2/2     Running     0               21h     10.16.4.31    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/node-local-dns-4s4qp                                             1/1     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/node-local-dns-5thmh                                             1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/node-local-dns-m8wnr                                             1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/pdcsi-node-c7h6d                                                 2/2     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/pdcsi-node-jd8rg                                                 2/2     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/pdcsi-node-jvmst                                                 2/2     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      
      NAMESPACE        NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP              PORT(S)                      AGE      SELECTOR
      cert-manager     service/cert-manager                            ClusterIP      10.20.3.107    <none>                   9402/TCP                     98d      app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
      cert-manager     service/cert-manager-webhook                    ClusterIP      10.20.1.42     <none>                   443/TCP                      98d      app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
      custom-metrics   service/custom-metrics-stackdriver-adapter      ClusterIP      10.20.6.153    <none>                   443/TCP                      97d      k8s-app=custom-metrics-stackdriver-adapter,run=custom-metrics-stackdriver-adapter
      default          service/kubernetes                              ClusterIP      10.20.0.1      <none>                   443/TCP                      3y58d    <none>
      default          service/mongodb-headless                        ClusterIP      None           <none>                   27017/TCP                    591d     app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb
      default          service/****-api                      ClusterIP      10.20.6.40     <none>                   6000/TCP                     2y283d   app=****-api
      default          service/****-extension-api            ClusterIP      10.20.14.28    <none>                   6000/TCP                     2y283d   app=****-extension-api
      default          service/****-extension-workers        ClusterIP      10.20.13.241   <none>                   6000/TCP                     2y283d   app=****-extension-workers
      default          service/****-extension-workers-next   ClusterIP      10.20.6.253    <none>                   6000/TCP                     286d     app=****-extension-workers-next
      default          service/****-renderer                 ClusterIP      10.20.15.100   <none>                   6100/TCP                     368d     app=****-renderer
      default          service/****-spa                      ClusterIP      10.20.0.127    <none>                   80/TCP                       2y270d   app=****-spa
      default          service/****-storage                  ExternalName   <none>         storage.googleapis.com   <none>                       552d     <none>
      default          service/****-webhook-api              ClusterIP      10.20.4.248    <none>                   6000/TCP                     79d      app=****-webhook-api
      default          service/redis                                   ClusterIP      10.20.12.99    <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/component=node,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
      default          service/redis-headless                          ClusterIP      None           <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
      default          service/redis-node                              ClusterIP      10.20.15.109   <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/component=node,app.kubernetes.io/instance=redis-node,app.kubernetes.io/name=redis
      default          service/redis-node-headless                     ClusterIP      None           <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/instance=redis-node,app.kubernetes.io/name=redis
      ingress-nginx    service/ingress-nginx-controller                LoadBalancer   10.20.15.191   35.193.145.228           80:32447/TCP,443:30526/TCP   25m      app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      ingress-nginx    service/ingress-nginx-controller-admission      ClusterIP      10.20.6.148    <none>                   443/TCP                      25m      app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      kube-system      service/kube-dns                                ClusterIP      10.20.0.10     <none>                   53/UDP,53/TCP                3y58d    k8s-app=kube-dns
      kube-system      service/kube-dns-upstream                       ClusterIP      10.20.9.195    <none>                   53/UDP,53/TCP                5d22h    k8s-app=kube-dns
      kube-system      service/metrics-server                          ClusterIP      10.20.5.201    <none>                   443/TCP                      3y58d    k8s-app=metrics-server
      
      NAMESPACE     NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE     CONTAINERS                                 IMAGES                                                                                                                SELECTOR
      kube-system   daemonset.apps/fluentbit-gke                  3         3         3       3            3           kubernetes.io/os=linux                                               2y      fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/fluentbit-gke-256pd            0         0         0       0            0           kubernetes.io/os=linux                                               95d     fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/fluentbit-gke-max              0         0         0       0            0           kubernetes.io/os=linux                                               60d     fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/gke-metrics-agent              3         3         3       3            3           <none>                                                               397d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-scaling-10   0         0         0       0            0           <none>                                                               128d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-scaling-20   0         0         0       0            0           <none>                                                               128d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-windows      0         0         0       0            0           kubernetes.io/os=windows                                             2y23d   gke-metrics-agent                          gke.gcr.io/gke-metrics-agent-windows:1.8.3-gke.0                                                                      component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/kube-proxy                     0         0         0       0            0           kubernetes.io/os=linux,node.kubernetes.io/kube-proxy-ds-ready=true   2y23d   kube-proxy                                 gke.gcr.io/kube-proxy-amd64:v1.22.12-gke.2300                                                                         k8s-app=kube-proxy
      kube-system   daemonset.apps/metadata-proxy-v0.1            0         0         0       0            0           cloud.google.com/metadata-proxy-ready=true,kubernetes.io/os=linux    179d    metadata-proxy,prometheus-to-sd-exporter   k8s.gcr.io/metadata-proxy:v0.1.12,gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1                                           k8s-app=metadata-proxy,version=v0.1
      kube-system   daemonset.apps/node-local-dns                 3         3         3       3            3           addon.gke.io/node-local-dns-ds-ready=true                            5d22h   node-cache                                 gke.gcr.io/k8s-dns-node-cache:1.22.2-gke.0                                                                            k8s-app=node-local-dns
      kube-system   daemonset.apps/nvidia-gpu-device-plugin       0         0         0       0            0           <none>                                                               621d    nvidia-gpu-device-plugin                   gcr.io/gke-release/nvidia-gpu-device-plugin@sha256:80a976aecbc2c4cd4bae934807474bcd6f36baed94d4c122a52f87c04152e060   k8s-app=nvidia-gpu-device-plugin
      kube-system   daemonset.apps/pdcsi-node                     3         3         3       3            3           kubernetes.io/os=linux                                               598d    csi-driver-registrar,gce-pd-driver         gke.gcr.io/csi-node-driver-registrar:v2.5.0-gke.1,gke.gcr.io/gcp-compute-persistent-disk-csi-driver:v1.7.3-gke.1      k8s-app=gcp-compute-persistent-disk-csi-driver
      kube-system   daemonset.apps/pdcsi-node-windows             0         0         0       0            0           kubernetes.io/os=windows                                             535d    csi-driver-registrar,gce-pd-driver         gke.gcr.io/csi-node-driver-registrar:v2.5.0-gke.1,gke.gcr.io/gcp-compute-persistent-disk-csi-driver:v1.7.3-gke.1      k8s-app=gcp-compute-persistent-disk-csi-driver
      
      NAMESPACE        NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE      CONTAINERS                                 IMAGES                                                                                                                                                                     SELECTOR
      cert-manager     deployment.apps/cert-manager                         1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-controller:v1.9.1                                                                                                                            app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
      cert-manager     deployment.apps/cert-manager-cainjector              1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-cainjector:v1.9.1                                                                                                                            app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector
      cert-manager     deployment.apps/cert-manager-webhook                 1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-webhook:v1.9.1                                                                                                                               app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
      custom-metrics   deployment.apps/custom-metrics-stackdriver-adapter   1/1     1            1           97d      pod-custom-metrics-stackdriver-adapter     gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0                                                                                                        k8s-app=custom-metrics-stackdriver-adapter,run=custom-metrics-stackdriver-adapter
      default          deployment.apps/****-api                   4/4     4            4           2y282d   ****-api                         gcr.io/****/api/main@sha256:a41a8cc6d249188c9145f43e4432ad0f966752d164fa8b637d1782c1724f739a                                                                     app=****-api
      default          deployment.apps/****-extension-api         4/4     4            4           2y283d   ****-extension-api               gcr.io/****/extension-api/main@sha256:4907584f47f2e1290d2e47027005b52759cdfa253d43ae4db06d1e94444ce5ab                                                           app=****-extension-api
      default          deployment.apps/****-extension-workers     6/6     6            6           2y38d    ****-extension-workers           gcr.io/****/extension-workers/main@sha256:5605c7334f73e6ff87f116fe39ef7da1a2052f9cbbab6225664688b99db1caed                                                       app=****-extension-workers
      default          deployment.apps/****-renderer              1/1     1            1           368d     ****-renderer                    gcr.io/****/renderer/main@sha256:2d72651a9393d8f5a701d18968b8cd88d55c385fd926c83922e090bcbf4f87cf                                                                app=****-renderer
      default          deployment.apps/****-webhook-api           2/2     2            2           79d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:eec95ede41e11cdb21309f085874d8fc2f273580d1457ac2d63d4c61cad35229                                                             app=****-webhook-api
      ingress-nginx    deployment.apps/ingress-nginx-controller             1/1     1            1           25m      controller                                 registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143                                                    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      kube-system      deployment.apps/event-exporter-gke                   1/1     1            1           2y23d    event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.9-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                           k8s-app=event-exporter
      kube-system      deployment.apps/konnectivity-agent                   3/3     3            3           362d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.31-gke.0                                                                                                                                       k8s-app=konnectivity-agent
      kube-system      deployment.apps/konnectivity-agent-autoscaler        1/1     1            1           362d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                     k8s-app=konnectivity-agent-autoscaler
      kube-system      deployment.apps/kube-dns                             2/2     2            2           3y58d    kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.3-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0   k8s-app=kube-dns
      kube-system      deployment.apps/kube-dns-autoscaler                  1/1     1            1           3y58d    autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                     k8s-app=kube-dns-autoscaler
      kube-system      deployment.apps/metrics-server-v0.4.5                1/1     1            1           236d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                         k8s-app=metrics-server,version=v0.4.5
      
      NAMESPACE        NAME                                                            DESIRED   CURRENT   READY   AGE      CONTAINERS                                 IMAGES                                                                                                                                                                                   SELECTOR
      cert-manager     replicaset.apps/cert-manager-7b8d75c477                         1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-controller:v1.9.1                                                                                                                                          app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=7b8d75c477
      cert-manager     replicaset.apps/cert-manager-cainjector-6cd8d7f84b              1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-cainjector:v1.9.1                                                                                                                                          app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=6cd8d7f84b
      cert-manager     replicaset.apps/cert-manager-webhook-64d76db6c                  1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-webhook:v1.9.1                                                                                                                                             app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=64d76db6c
      custom-metrics   replicaset.apps/custom-metrics-stackdriver-adapter-848d58c7c7   1         1         1       97d      pod-custom-metrics-stackdriver-adapter     gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0                                                                                                                      k8s-app=custom-metrics-stackdriver-adapter,pod-template-hash=848d58c7c7,run=custom-metrics-stackdriver-adapter
      default          replicaset.apps/****-api-5488fb47c6                   0         0         0       15d      ****-api                         gcr.io/****/api/main@sha256:1439a994421ab2103ead8f8b7ac46d94eea8b2780a67eef7f07a7f3bf1a04ab3                                                                                   app=****-api,pod-template-hash=5488fb47c6
      default          replicaset.apps/****-api-586567774                    0         0         0       24d      ****-api                         gcr.io/****/api/main@sha256:dec6793c14b20db901826722befe7dff97a04099d275381098273d3b1bc7a64e                                                                                   app=****-api,pod-template-hash=586567774
      default          replicaset.apps/****-api-5876bb65                     0         0         0       4d22h    ****-api                         gcr.io/****/api/main@sha256:7c8cb3bb4e0ae7315ee5efb0676e7a9d3b594d04b6776cc443072f0af9aa76d0                                                                                   app=****-api,pod-template-hash=5876bb65
      default          replicaset.apps/****-api-69d6569587                   0         0         0       35d      ****-api                         gcr.io/****/api/main@sha256:b5f501548efc2eda05044db36cb6c93ca117f5a85962b13db1f2b09f12fe1a30                                                                                   app=****-api,pod-template-hash=69d6569587
      default          replicaset.apps/****-api-6d7b645f44                   0         0         0       39d      ****-api                         gcr.io/****/api/main@sha256:d6954db7435f091b52a06f39b0886399fecdb43e3bc7c8551d6fe067d3dcb22f                                                                                   app=****-api,pod-template-hash=6d7b645f44
      default          replicaset.apps/****-api-6fcfff547                    0         0         0       20d      ****-api                         gcr.io/****/api/main@sha256:b4a0534cf5a502eabba1aa61370a452068cf0704d0f4c3ff9b7a242cff3aab89                                                                                   app=****-api,pod-template-hash=6fcfff547
      default          replicaset.apps/****-api-77b785ff59                   0         0         0       35d      ****-api                         gcr.io/****/api/main@sha256:d65ab2c2694c18ed5806506b7a76cb641fc04071d0362cecec6a4e451c531a6f                                                                                   app=****-api,pod-template-hash=77b785ff59
      default          replicaset.apps/****-api-7b5c9dd98b                   4         4         4       3d22h    ****-api                         gcr.io/****/api/main@sha256:a41a8cc6d249188c9145f43e4432ad0f966752d164fa8b637d1782c1724f739a                                                                                   app=****-api,pod-template-hash=7b5c9dd98b
      default          replicaset.apps/****-api-7f987f4b84                   0         0         0       27d      ****-api                         gcr.io/****/api/main@sha256:3f35a92f97d417cefb54ab7221442b3c5ff64339f32c8f32fb3dd3b2feb69b9b                                                                                   app=****-api,pod-template-hash=7f987f4b84
      default          replicaset.apps/****-api-df967849c                    0         0         0       23d      ****-api                         gcr.io/****/api/main@sha256:b1e53c55254b319d6f0232106a444570d8dab0facddb214dc84e185d79d46cc8                                                                                   app=****-api,pod-template-hash=df967849c
      default          replicaset.apps/****-api-fb656fc69                    0         0         0       24d      ****-api                         gcr.io/****/api/main@sha256:01544143593b8096fc3c4a61de210ba01b7a4e2de8bd5c9fef69c8dad896bcbf                                                                                   app=****-api,pod-template-hash=fb656fc69
      default          replicaset.apps/****-extension-api-5685d8c768         0         0         0       72d      ****-extension-api               gcr.io/****/extension-api/main@sha256:b7514236bcc5330e053a3d08df3ebb3e43b2153f8ed23fe1f57cf9c87bc2fe13                                                                         app=****-extension-api,pod-template-hash=5685d8c768
      default          replicaset.apps/****-extension-api-59dcd6d7cc         0         0         0       35d      ****-extension-api               gcr.io/****/extension-api/main@sha256:922d644fd379eaafa64c82a2b8c1b81efc49f896c96a2b8497f4cb59a728c0c4                                                                         app=****-extension-api,pod-template-hash=59dcd6d7cc
      default          replicaset.apps/****-extension-api-5fd9dd57d6         0         0         0       35d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1dd2368f94dce47c9acea2756f0b862e66ca116d9bedde0a10542911d8b05210                                                                         app=****-extension-api,pod-template-hash=5fd9dd57d6
      default          replicaset.apps/****-extension-api-64964c8b7          4         4         4       15d      ****-extension-api               gcr.io/****/extension-api/main@sha256:4907584f47f2e1290d2e47027005b52759cdfa253d43ae4db06d1e94444ce5ab                                                                         app=****-extension-api,pod-template-hash=64964c8b7
      default          replicaset.apps/****-extension-api-654db46b5f         0         0         0       70d      ****-extension-api               gcr.io/****/extension-api/main@sha256:3703ef5cacbb442cc9f65a7f028bf21a28a082b38a938892b5ca0abe15e40ecc                                                                         app=****-extension-api,pod-template-hash=654db46b5f
      default          replicaset.apps/****-extension-api-6559bb9d55         0         0         0       56d      ****-extension-api               gcr.io/****/extension-api/main@sha256:7cb4b299427172e3c0f3da6d908a5e7c64a338c9773e9696357b2b8a23d040ad                                                                         app=****-extension-api,pod-template-hash=6559bb9d55
      default          replicaset.apps/****-extension-api-676dcdcdf4         0         0         0       77d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1f807cf9d0aa5f8eaa6a078b6bcb0c4a2474641e18942b4f6de0f2f240e4454e                                                                         app=****-extension-api,pod-template-hash=676dcdcdf4
      default          replicaset.apps/****-extension-api-74764fc5b9         0         0         0       24d      ****-extension-api               gcr.io/****/extension-api/main@sha256:26c01c24cf2a9a156d12b40df0106174e6f04c482bf83d2314c734d604bb927d                                                                         app=****-extension-api,pod-template-hash=74764fc5b9
      default          replicaset.apps/****-extension-api-757d8bdc5c         0         0         0       24d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1e647caa6569e6a7a047be50b2cd3c40a18e9fd71ca334c8b8c0460b83516244                                                                         app=****-extension-api,pod-template-hash=757d8bdc5c
      default          replicaset.apps/****-extension-api-85d9dd68bd         0         0         0       23d      ****-extension-api               gcr.io/****/extension-api/main@sha256:8a4c00850c8719f4f6cf5a33fd400898de91b3fda421220a374aa1663e928fb4                                                                         app=****-extension-api,pod-template-hash=85d9dd68bd
      default          replicaset.apps/****-extension-api-cf6cb5d5c          0         0         0       27d      ****-extension-api               gcr.io/****/extension-api/main@sha256:62fb63417fdec37d77f899db4e49f53ff09cc0a4d7698d3a490713f2b923fcc8                                                                         app=****-extension-api,pod-template-hash=cf6cb5d5c
      default          replicaset.apps/****-extension-workers-548cb46fdc     0         0         0       23d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:d1e8ca29ca2211638fa75c722e83691f4d0414ce1ccbfc6e2ddc57135ea26c1b                                                                     app=****-extension-workers,pod-template-hash=548cb46fdc
      default          replicaset.apps/****-extension-workers-5b66577498     0         0         0       21d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:783ab87a076616c3da7db42f3847dbd46354495c046213736a1a84ca348df7d5                                                                     app=****-extension-workers,pod-template-hash=5b66577498
      default          replicaset.apps/****-extension-workers-5c847c5456     6         6         6       15d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:5605c7334f73e6ff87f116fe39ef7da1a2052f9cbbab6225664688b99db1caed                                                                     app=****-extension-workers,pod-template-hash=5c847c5456
      default          replicaset.apps/****-extension-workers-664b88db8c     0         0         0       25d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:4b3af3c696cf9e539f2697f0c338f804aa8019177000143302e38fb2bf5c574c                                                                     app=****-extension-workers,pod-template-hash=664b88db8c
      default          replicaset.apps/****-extension-workers-68cbd968cc     0         0         0       18d      ****-extension-workers           gcr.io/****/extension-workers/pull/554@sha256:3b17e84bb29fad14e009496288de8954caa314f8fc0874d925eae6c2616b3af3                                                                 app=****-extension-workers,pod-template-hash=68cbd968cc
      default          replicaset.apps/****-extension-workers-6f5854878      0         0         0       25d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:8f752bf2fa4ecb3d040d9a345a1985b554b05c3b2fe570ebc2e818f3cfb68059                                                                     app=****-extension-workers,pod-template-hash=6f5854878
      default          replicaset.apps/****-extension-workers-76c48dd85c     0         0         0       23d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:2d2bcdd4ebac4d69607e32372b95f2994ba46ad1be49c2396e2a878e73bb05ea                                                                     app=****-extension-workers,pod-template-hash=76c48dd85c
      default          replicaset.apps/****-extension-workers-76d9d5b69c     0         0         0       27d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:b933924ef2ab0c36c640d15d0449c7cf63e147a6cb0f68f170e4e74a176d3e06                                                                     app=****-extension-workers,pod-template-hash=76d9d5b69c
      default          replicaset.apps/****-extension-workers-799fd4b667     0         0         0       20d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:8d2327560cdc3d7011d8b53e52dbd364ba23b179c0f0b48df4d53db5c68decef                                                                     app=****-extension-workers,pod-template-hash=799fd4b667
      default          replicaset.apps/****-extension-workers-7d7bc97d65     0         0         0       24d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:7421e698772b4c740f792b41ca1ad82a01d4a87f608330a42ba5a4ebd5ee31c9                                                                     app=****-extension-workers,pod-template-hash=7d7bc97d65
      default          replicaset.apps/****-extension-workers-8647d57898     0         0         0       24d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:88275fac722a7d42314dcd9f0a757e7c0db461855b3b56f2226400300120b7e9                                                                     app=****-extension-workers,pod-template-hash=8647d57898
      default          replicaset.apps/****-renderer-54cd644656              0         0         0       56d      ****-renderer                    gcr.io/****/renderer/main@sha256:07a6585e5238e2d63df117c18ecd9b8d5a57fbfa9765b6babb0707d56b845246                                                                              app=****-renderer,pod-template-hash=54cd644656
      default          replicaset.apps/****-renderer-5589dc84d9              0         0         0       35d      ****-renderer                    gcr.io/****/renderer/main@sha256:90fc79630921b7a67b86df450c5ae386994e3b2276bd911c83b8c6e4bd1943b4                                                                              app=****-renderer,pod-template-hash=5589dc84d9
      default          replicaset.apps/****-renderer-58db5ddd8d              0         0         0       24d      ****-renderer                    gcr.io/****/renderer/main@sha256:bfed03c7bfc07153a4c7f382de61f67ada23647a25d32e0122689aa9019f549a                                                                              app=****-renderer,pod-template-hash=58db5ddd8d
      default          replicaset.apps/****-renderer-5d4464b756              0         0         0       77d      ****-renderer                    gcr.io/****/renderer/main@sha256:228364867a0a645b64bcd10fcb2c2c1c5f060928eb2c7fb0989f718d879fbcf4                                                                              app=****-renderer,pod-template-hash=5d4464b756
      default          replicaset.apps/****-renderer-5ff77dd894              0         0         0       24d      ****-renderer                    gcr.io/****/renderer/main@sha256:4004c35f60c0efe80a58aeafbe534a5bd77ec8c261c96fec167a509d2ae31fe8                                                                              app=****-renderer,pod-template-hash=5ff77dd894
      default          replicaset.apps/****-renderer-6554bfdf4               0         0         0       35d      ****-renderer                    gcr.io/****/renderer/main@sha256:a9b42e5d92092ece7b120406ca07aa01efef5c11461baf4eaf25182b4afd2c61                                                                              app=****-renderer,pod-template-hash=6554bfdf4
      default          replicaset.apps/****-renderer-66959b6dc5              1         1         1       15d      ****-renderer                    gcr.io/****/renderer/main@sha256:2d72651a9393d8f5a701d18968b8cd88d55c385fd926c83922e090bcbf4f87cf                                                                              app=****-renderer,pod-template-hash=66959b6dc5
      default          replicaset.apps/****-renderer-76b847c999              0         0         0       27d      ****-renderer                    gcr.io/****/renderer/main@sha256:e8d2916a5929dfeee8748fe77b357d658b80408712b1754123be24f8d778b01f                                                                              app=****-renderer,pod-template-hash=76b847c999
      default          replicaset.apps/****-renderer-7b57bd4d98              0         0         0       72d      ****-renderer                    gcr.io/****/renderer/main@sha256:b83b0f24397d8e0127f8dc3616fe257bd2178acd3d841eb02a5d98442dbdc334                                                                              app=****-renderer,pod-template-hash=7b57bd4d98
      default          replicaset.apps/****-renderer-85866fbc7c              0         0         0       70d      ****-renderer                    gcr.io/****/renderer/main@sha256:faab61a914cefd4d24537a43886eced978b4cf6bd486425a4a1cf1ee02d2e865                                                                              app=****-renderer,pod-template-hash=85866fbc7c
      default          replicaset.apps/****-renderer-d58fb68b5               0         0         0       23d      ****-renderer                    gcr.io/****/renderer/main@sha256:2d80b6c3b8cee7ba3f45eab53b83f75954c068b732ba07f39cd681129a6c3971                                                                              app=****-renderer,pod-template-hash=d58fb68b5
      default          replicaset.apps/****-webhook-api-58946cd95b           0         0         0       35d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:b2a3965ed89f284cc54fc7b6b51e2d679ad2889649c992d01794dd4dbc2e8e1a                                                                           app=****-webhook-api,pod-template-hash=58946cd95b
      default          replicaset.apps/****-webhook-api-5fb64b4958           0         0         0       23d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:4c699a28570bf689d512361be8f1260a43c570d062a1491f2d00116094ad85fa                                                                           app=****-webhook-api,pod-template-hash=5fb64b4958
      default          replicaset.apps/****-webhook-api-5fcfd5d96b           0         0         0       70d      ****-webhook-api                 gcr.io/****/webhook-api/pull/527@sha256:bc1971939772069843611afbcb389d7d27e793d25d8311860f973611640099aa                                                                       app=****-webhook-api,pod-template-hash=5fcfd5d96b
      default          replicaset.apps/****-webhook-api-686475565c           0         0         0       56d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:001c1293692dc38c810d2440ce5a4bdf99db65339724250d8f73b265aa8aa958                                                                           app=****-webhook-api,pod-template-hash=686475565c
      default          replicaset.apps/****-webhook-api-6f7bdf6d99           0         0         0       24d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:9882451f7a39ad9ec5d0a31fb3f7f4bef34666bab58e0305981e5100ba44852f                                                                           app=****-webhook-api,pod-template-hash=6f7bdf6d99
      default          replicaset.apps/****-webhook-api-6f8db6fb             0         0         0       24d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:67fcfb97e04fd26e77ebf8e48310a29e0441d9ea7ed04524e41d27cdb80ed2f9                                                                           app=****-webhook-api,pod-template-hash=6f8db6fb
      default          replicaset.apps/****-webhook-api-6fcd976b57           0         0         0       27d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:11390b9ac2b2800c522fc89afacee26798b6e022028a0506e18a004e55fc33f1                                                                           app=****-webhook-api,pod-template-hash=6fcd976b57
      default          replicaset.apps/****-webhook-api-7776866645           0         0         0       52d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:b2a6a4d81a6eae7b728e7fb0a217f3196684dc74caefb04bef3c9ffa1cffe5e7                                                                           app=****-webhook-api,pod-template-hash=7776866645
      default          replicaset.apps/****-webhook-api-86bb7d96fd           2         2         2       15d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:eec95ede41e11cdb21309f085874d8fc2f273580d1457ac2d63d4c61cad35229                                                                           app=****-webhook-api,pod-template-hash=86bb7d96fd
      default          replicaset.apps/****-webhook-api-979cd4fc9            0         0         0       70d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:646941a68e7691448e43cee142daebe071c14f62af8986c2f0ef427be5ba4797                                                                           app=****-webhook-api,pod-template-hash=979cd4fc9
      default          replicaset.apps/****-webhook-api-cc65bb7b6            0         0         0       35d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:dade0020a9c87760588783094fa608b688d00149c122bc6e9f1fa5724f5df24b                                                                           app=****-webhook-api,pod-template-hash=cc65bb7b6
      ingress-nginx    replicaset.apps/ingress-nginx-controller-69fbfb4bfd             1         1         1       25m      controller                                 registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143                                                                  app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=69fbfb4bfd
      kube-system      replicaset.apps/event-exporter-gke-5479fd58c8                   0         0         0       236d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.5-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=5479fd58c8
      kube-system      replicaset.apps/event-exporter-gke-666b7ffbf7                   0         0         0       663d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=666b7ffbf7
      kube-system      replicaset.apps/event-exporter-gke-77cccd97c6                   0         0         0       696d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=77cccd97c6
      kube-system      replicaset.apps/event-exporter-gke-8489df9489                   0         0         0       2y23d    event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=8489df9489
      kube-system      replicaset.apps/event-exporter-gke-f66d9f855                    1         1         1       3d5h     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.9-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=f66d9f855
      kube-system      replicaset.apps/konnectivity-agent-54dcfd5b75                   0         0         0       235d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=54dcfd5b75
      kube-system      replicaset.apps/konnectivity-agent-68dbdb55                     0         0         0       102d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.27-gke.2                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=68dbdb55
      kube-system      replicaset.apps/konnectivity-agent-6fb49c9c54                   0         0         0       166d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=6fb49c9c54
      kube-system      replicaset.apps/konnectivity-agent-7765499767                   0         0         0       236d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=7765499767
      kube-system      replicaset.apps/konnectivity-agent-7998ff9dd6                   0         0         0       362d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.22-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=7998ff9dd6
      kube-system      replicaset.apps/konnectivity-agent-8484759759                   3         3         3       38d      konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.31-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=8484759759
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-5c49cb58bb        0         0         0       236d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=5c49cb58bb
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6b86f667c9        0         0         0       166d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6b86f667c9
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6cb774c9cc        0         0         0       362d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6cb774c9cc
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6dfb4f9cfb        1         1         1       38d      autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6dfb4f9cfb
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-7fd5dd4f5         0         0         0       102d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=7fd5dd4f5
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-ddccb8b95         0         0         0       235d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=ddccb8b95
      kube-system      replicaset.apps/kube-dns-56494768b7                             0         0         0       102d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.21.4-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.21.4-gke.0,gke.gcr.io/k8s-dns-sidecar:1.21.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=56494768b7
      kube-system      replicaset.apps/kube-dns-5d54b45645                             0         0         0       535d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.17.1-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.17.1-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.17.1-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2      k8s-app=kube-dns,pod-template-hash=5d54b45645
      kube-system      replicaset.apps/kube-dns-6465f78586                             0         0         0       572d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=6465f78586
      kube-system      replicaset.apps/kube-dns-66d6b7c877                             0         0         0       663d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.15-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.15-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.15-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=66d6b7c877
      kube-system      replicaset.apps/kube-dns-6bd88c9b66                             0         0         0       621d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=6bd88c9b66
      kube-system      replicaset.apps/kube-dns-6c7b8dc9f9                             0         0         0       506d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.17.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.17.3-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.17.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2      k8s-app=kube-dns,pod-template-hash=6c7b8dc9f9
      kube-system      replicaset.apps/kube-dns-7f4d6f474d                             0         0         0       270d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.21.0-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.21.0-gke.0,gke.gcr.io/k8s-dns-sidecar:1.21.0-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2                        k8s-app=kube-dns,pod-template-hash=7f4d6f474d
      kube-system      replicaset.apps/kube-dns-85df8994db                             0         0         0       94d      kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.2-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.2-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.2-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=85df8994db
      kube-system      replicaset.apps/kube-dns-9c59558bb                              0         0         0       642d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=9c59558bb
      kube-system      replicaset.apps/kube-dns-autoscaler-58cbd4f75c                  0         0         0       535d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=58cbd4f75c
      kube-system      replicaset.apps/kube-dns-autoscaler-5c78d65cd9                  0         0         0       2y       autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=5c78d65cd9
      kube-system      replicaset.apps/kube-dns-autoscaler-645f7d66cf                  0         0         0       2y23d    autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=645f7d66cf
      kube-system      replicaset.apps/kube-dns-autoscaler-76fcd5f658                  0         0         0       3y58d    autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.2.0                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=76fcd5f658
      kube-system      replicaset.apps/kube-dns-autoscaler-7f89fb6b79                  0         0         0       621d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=7f89fb6b79
      kube-system      replicaset.apps/kube-dns-autoscaler-844c9d9448                  0         0         0       460d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=844c9d9448
      kube-system      replicaset.apps/kube-dns-autoscaler-85f8bdb54                   0         0         0       2y314d   autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=85f8bdb54
      kube-system      replicaset.apps/kube-dns-autoscaler-8687c64fc                   0         0         0       2y238d   autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=8687c64fc
      kube-system      replicaset.apps/kube-dns-autoscaler-bb58c6784                   0         0         0       2y314d   autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.3.0                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=bb58c6784
      kube-system      replicaset.apps/kube-dns-autoscaler-f4d55555                    1         1         1       102d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=f4d55555
      kube-system      replicaset.apps/kube-dns-b4f5c58c7                              0         0         0       418d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.19.1-gke.1,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.19.1-gke.1,gke.gcr.io/k8s-dns-sidecar:1.19.1-gke.1,gke.gcr.io/prometheus-to-sd:v0.4.2                        k8s-app=kube-dns,pod-template-hash=b4f5c58c7
      kube-system      replicaset.apps/kube-dns-b99c7865d                              2         2         2       44d      kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.3-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=b99c7865d
      kube-system      replicaset.apps/metrics-server-v0.4.5-56fcb96597                0         0         0       236d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=56fcb96597,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-57bfdff4dd                0         0         0       128d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=57bfdff4dd,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-5bdb996c9f                0         0         0       232d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=5bdb996c9f,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-68f97d654f                0         0         0       153d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=68f97d654f,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-788d9b568b                0         0         0       88d      metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=788d9b568b,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-bbb794dcc                 0         0         0       236d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=bbb794dcc,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-fb4c49dd6                 1         1         1       126d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=fb4c49dd6,version=v0.4.5
      
      NAMESPACE   NAME                          READY   AGE    CONTAINERS       IMAGES
      default     statefulset.apps/mongodb      3/3     591d   mongodb          docker.io/bitnami/mongodb:4.4.4-debian-10-r30
      default     statefulset.apps/redis-node   3/3     433d   redis,sentinel   docker.io/bitnami/redis:6.2.5-debian-10-r34,docker.io/bitnami/redis-sentinel:6.2.5-debian-10-r34
      
      NAMESPACE   NAME                                                                   REFERENCE                                     TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
      default     horizontalpodautoscaler.autoscaling/****-extension-workers   Deployment/****-extension-workers   80666m/75   4         40        6          98d
      
    • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
      NAMESPACE        NAME                                                                 READY   STATUS      RESTARTS        AGE     IP            NODE                                                  NOMINATED NODE   READINESS GATES
      cert-manager     pod/cert-manager-7b8d75c477-55nn9                                    1/1     Running     2 (3d5h ago)    5d22h   10.16.4.2     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      cert-manager     pod/cert-manager-cainjector-6cd8d7f84b-swll8                         1/1     Running     10 (3d5h ago)   5d22h   10.16.0.2     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      cert-manager     pod/cert-manager-webhook-64d76db6c-t568k                             1/1     Running     0               5d21h   10.16.0.14    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      cert-manager     pod/cert-manager-webhook-64d76db6c-ttd66                             0/1     Completed   0               5d22h   10.16.5.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      custom-metrics   pod/custom-metrics-stackdriver-adapter-848d58c7c7-zjjwb              1/1     Running     0               5d22h   10.16.0.3     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/mongodb-0                                                        1/1     Running     0               5d22h   10.16.4.10    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/mongodb-1                                                        1/1     Running     0               5d22h   10.16.0.11    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/mongodb-2                                                        1/1     Running     0               5d22h   10.16.4.12    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-api-7b5c9dd98b-24nff                              1/1     Running     0               3d22h   10.16.4.24    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-api-7b5c9dd98b-2bdxj                              1/1     Running     0               3d22h   10.16.0.23    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-api-7b5c9dd98b-5gp2g                              1/1     Running     0               3d22h   10.16.0.24    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-api-7b5c9dd98b-tc5p2                              1/1     Running     1 (3d22h ago)   3d22h   10.16.4.23    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-25qv7                     1/1     Running     4 (5d22h ago)   5d22h   10.16.4.6     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-67s99                     1/1     Running     0               5d22h   10.16.0.8     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-extension-api-64964c8b7-bnqvr                     1/1     Running     4 (5d22h ago)   5d22h   10.16.4.5     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-extension-api-64964c8b7-vrfc7                     1/1     Running     0               5d22h   10.16.0.5     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5g54z                0/1     Error       0               5d5h    10.16.5.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5prjr                0/1     Error       0               3d21h   10.16.1.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-5s4t6                0/1     Error       1 (3d23h ago)   3d23h   10.16.1.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-6sslf                0/1     Error       0               23h     10.16.1.17    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-77dtz                1/1     Running     0               20h     10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-7mqg4                0/1     Error       0               4d22h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-7tjfg                0/1     Error       0               45h     10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-857rm                1/1     Running     0               54m     10.16.1.19    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-8lcdn                0/1     Error       0               3d21h   10.16.1.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-8x49l                0/1     Error       0               2d22h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-9nn5c                0/1     Error       0               3d21h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bbsvn                0/1     Error       0               5d21h   10.16.5.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bpsmq                0/1     Error       0               3d21h   10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-bz5k5                0/1     Error       0               45h     10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-c6rg2                0/1     Error       0               4d23h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-ccp9c                0/1     Error       0               47h     10.16.1.11    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-cqbsk                0/1     Error       0               5d      10.16.5.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dbs7b                0/1     Error       0               2d20h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dfh6c                0/1     Error       0               3d      10.16.1.16    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dj6vk                0/1     Error       0               2d22h   10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dtkm2                0/1     Error       0               3d21h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-dv5kz                0/1     Error       0               5d21h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-fm9jk                0/1     Error       0               5d22h   10.16.5.11    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-g8gvm                1/1     Running     0               4h18m   10.16.1.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-gpv72                0/1     Error       0               3d21h   10.16.1.3     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-gxl9c                0/1     Error       0               45h     10.16.1.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-hmcst                0/1     Error       0               45h     10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-hq7nt                0/1     Error       0               3d21h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-jqpqr                0/1     Error       0               5d21h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kfnpr                0/1     Error       0               4d22h   10.16.1.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kjbvc                0/1     Error       0               5d22h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-kt2tv                0/1     Error       0               5d5h    10.16.5.14    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-lm46m                0/1     Error       0               2d22h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-lmrnn                0/1     Error       0               23h     10.16.1.18    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-mm6xt                0/1     Error       0               2d23h   10.16.1.17    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-nb8qj                0/1     Error       0               2d22h   10.16.1.18    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-ncpjh                0/1     Error       0               3d21h   10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-nfvpn                0/1     Error       0               3d21h   10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-q7m4q                0/1     Error       0               4d22h   10.16.5.7     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-qxwb7                0/1     Error       0               5d22h   10.16.5.13    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rgjxc                1/1     Running     0               19h     10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rn89g                0/1     Error       0               4d22h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rrcws                0/1     Error       0               2d19h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-rtlbd                1/1     Running     0               20h     10.16.1.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-sgljb                0/1     Error       0               4d23h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-szjh7                0/1     Error       0               4d22h   10.16.5.8     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-t85jl                0/1     Error       0               3d21h   10.16.1.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-v56rf                0/1     Error       0               5d22h   10.16.5.12    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-v7qzv                0/1     Error       0               4d22h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-vnrk2                0/1     Error       0               2d22h   10.16.1.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wdh6j                0/1     Error       0               4d23h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wgzbw                0/1     Error       0               4d22h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-wqcdc                0/1     Error       0               45h     10.16.1.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-extension-workers-5c847c5456-zlcgq                1/1     Running     0               39m     10.16.1.23    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      default          pod/****-renderer-66959b6dc5-sqdmg                         1/1     Running     1 (5d22h ago)   5d22h   10.16.4.7     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/****-webhook-api-86bb7d96fd-6gxmv                      1/1     Running     0               5d22h   10.16.0.7     gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/****-webhook-api-86bb7d96fd-kqgxv                      1/1     Running     4 (5d22h ago)   5d22h   10.16.4.8     gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/redis-node-0                                                     2/2     Running     0               5d22h   10.16.0.13    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      default          pod/redis-node-1                                                     2/2     Running     0               5d22h   10.16.4.11    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      default          pod/redis-node-2                                                     2/2     Running     0               5d22h   10.16.4.13    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      ingress-nginx    pod/ingress-nginx-controller-69fbfb4bfd-k29tz                        1/1     Running     0               25m     10.16.1.25    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/event-exporter-gke-5479fd58c8-s44rs                              0/2     Completed   0               5d22h   10.16.5.9     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/event-exporter-gke-f66d9f855-8c7q2                               2/2     Running     0               2d22h   10.16.0.29    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/event-exporter-gke-f66d9f855-fqzvq                               0/2     Completed   0               3d5h    10.16.1.15    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/fluentbit-gke-dmnj9                                              2/2     Running     0               4d6h    10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/fluentbit-gke-kkmb2                                              2/2     Running     0               4d6h    10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/fluentbit-gke-qxwvq                                              2/2     Running     0               4d6h    10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/gke-metrics-agent-56dbf                                          1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/gke-metrics-agent-94xp8                                          1/1     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/gke-metrics-agent-ffz8h                                          1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-5hsqg                              1/1     Running     0               18h     10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-8vxw7                              0/1     Error       0               5d22h   10.16.5.2     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-dhk5j                              0/1     Error       0               5d21h   10.16.5.4     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-hfc79                              0/1     Error       0               3d21h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-qk5p9                              0/1     Error       0               4d22h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-rnd6b                              0/1     Error       0               3d21h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-tfm5n                              1/1     Running     0               5d22h   10.16.0.12    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-wmjpj                              1/1     Running     0               5d21h   10.16.4.14    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-xp9jj                              0/1     Error       0               4d22h   10.16.5.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-xvm4m                              0/1     Error       0               45h     10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-8484759759-z2nsx                              0/1     Error       0               2d14h   10.16.1.10    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/konnectivity-agent-autoscaler-6dfb4f9cfb-5cd56                   1/1     Running     0               5d21h   10.16.0.16    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/konnectivity-agent-autoscaler-6dfb4f9cfb-mkl7h                   0/1     Error       0               5d22h   10.16.5.5     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-autoscaler-f4d55555-677qc                               0/1     Error       0               5d22h   10.16.5.6     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-autoscaler-f4d55555-npl4l                               1/1     Running     0               5d21h   10.16.0.15    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/kube-dns-b99c7865d-8bc5p                                         4/4     Running     0               5d22h   10.16.1.2     gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-dns-b99c7865d-rq5r7                                         4/4     Running     0               5d22h   10.16.0.10    gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/kube-proxy-gke-****-preemptive-pool-e2-e219fbd6-54gl   1/1     Running     0               3d22h   10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/kube-proxy-gke-****-standard-pool-e2-bae76619-3sf5     1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/kube-proxy-gke-****-standard-pool-e2-bae76619-ejn4     1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/metrics-server-v0.4.5-fb4c49dd6-tnp58                            2/2     Running     0               21h     10.16.4.31    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/node-local-dns-4s4qp                                             1/1     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/node-local-dns-5thmh                                             1/1     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/node-local-dns-m8wnr                                             1/1     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      kube-system      pod/pdcsi-node-c7h6d                                                 2/2     Running     0               20h     10.128.0.2    gke-****-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      kube-system      pod/pdcsi-node-jd8rg                                                 2/2     Running     0               5d22h   10.128.0.63   gke-****-standard-pool-e2-bae76619-ejn4     <none>           <none>
      kube-system      pod/pdcsi-node-jvmst                                                 2/2     Running     0               5d22h   10.128.0.5    gke-****-standard-pool-e2-bae76619-3sf5     <none>           <none>
      
      NAMESPACE        NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP              PORT(S)                      AGE      SELECTOR
      cert-manager     service/cert-manager                            ClusterIP      10.20.3.107    <none>                   9402/TCP                     98d      app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
      cert-manager     service/cert-manager-webhook                    ClusterIP      10.20.1.42     <none>                   443/TCP                      98d      app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
      custom-metrics   service/custom-metrics-stackdriver-adapter      ClusterIP      10.20.6.153    <none>                   443/TCP                      97d      k8s-app=custom-metrics-stackdriver-adapter,run=custom-metrics-stackdriver-adapter
      default          service/kubernetes                              ClusterIP      10.20.0.1      <none>                   443/TCP                      3y58d    <none>
      default          service/mongodb-headless                        ClusterIP      None           <none>                   27017/TCP                    591d     app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb
      default          service/****-api                      ClusterIP      10.20.6.40     <none>                   6000/TCP                     2y283d   app=****-api
      default          service/****-extension-api            ClusterIP      10.20.14.28    <none>                   6000/TCP                     2y283d   app=****-extension-api
      default          service/****-extension-workers        ClusterIP      10.20.13.241   <none>                   6000/TCP                     2y283d   app=****-extension-workers
      default          service/****-extension-workers-next   ClusterIP      10.20.6.253    <none>                   6000/TCP                     286d     app=****-extension-workers-next
      default          service/****-renderer                 ClusterIP      10.20.15.100   <none>                   6100/TCP                     368d     app=****-renderer
      default          service/****-spa                      ClusterIP      10.20.0.127    <none>                   80/TCP                       2y270d   app=****-spa
      default          service/****-storage                  ExternalName   <none>         storage.googleapis.com   <none>                       552d     <none>
      default          service/****-webhook-api              ClusterIP      10.20.4.248    <none>                   6000/TCP                     79d      app=****-webhook-api
      default          service/redis                                   ClusterIP      10.20.12.99    <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/component=node,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
      default          service/redis-headless                          ClusterIP      None           <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
      default          service/redis-node                              ClusterIP      10.20.15.109   <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/component=node,app.kubernetes.io/instance=redis-node,app.kubernetes.io/name=redis
      default          service/redis-node-headless                     ClusterIP      None           <none>                   6379/TCP,26379/TCP           433d     app.kubernetes.io/instance=redis-node,app.kubernetes.io/name=redis
      ingress-nginx    service/ingress-nginx-controller                LoadBalancer   10.20.15.191   35.193.145.228           80:32447/TCP,443:30526/TCP   25m      app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      ingress-nginx    service/ingress-nginx-controller-admission      ClusterIP      10.20.6.148    <none>                   443/TCP                      25m      app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      kube-system      service/kube-dns                                ClusterIP      10.20.0.10     <none>                   53/UDP,53/TCP                3y58d    k8s-app=kube-dns
      kube-system      service/kube-dns-upstream                       ClusterIP      10.20.9.195    <none>                   53/UDP,53/TCP                5d22h    k8s-app=kube-dns
      kube-system      service/metrics-server                          ClusterIP      10.20.5.201    <none>                   443/TCP                      3y58d    k8s-app=metrics-server
      
      NAMESPACE     NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE     CONTAINERS                                 IMAGES                                                                                                                SELECTOR
      kube-system   daemonset.apps/fluentbit-gke                  3         3         3       3            3           kubernetes.io/os=linux                                               2y      fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/fluentbit-gke-256pd            0         0         0       0            0           kubernetes.io/os=linux                                               95d     fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/fluentbit-gke-max              0         0         0       0            0           kubernetes.io/os=linux                                               60d     fluentbit,fluentbit-gke                    gke.gcr.io/fluent-bit:v1.8.7-gke.1,gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0                                   component=fluentbit-gke,k8s-app=fluentbit-gke
      kube-system   daemonset.apps/gke-metrics-agent              3         3         3       3            3           <none>                                                               397d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-scaling-10   0         0         0       0            0           <none>                                                               128d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-scaling-20   0         0         0       0            0           <none>                                                               128d    gke-metrics-agent                          gke.gcr.io/gke-metrics-agent:1.8.3-gke.0                                                                              component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/gke-metrics-agent-windows      0         0         0       0            0           kubernetes.io/os=windows                                             2y23d   gke-metrics-agent                          gke.gcr.io/gke-metrics-agent-windows:1.8.3-gke.0                                                                      component=gke-metrics-agent,k8s-app=gke-metrics-agent
      kube-system   daemonset.apps/kube-proxy                     0         0         0       0            0           kubernetes.io/os=linux,node.kubernetes.io/kube-proxy-ds-ready=true   2y23d   kube-proxy                                 gke.gcr.io/kube-proxy-amd64:v1.22.12-gke.2300                                                                         k8s-app=kube-proxy
      kube-system   daemonset.apps/metadata-proxy-v0.1            0         0         0       0            0           cloud.google.com/metadata-proxy-ready=true,kubernetes.io/os=linux    179d    metadata-proxy,prometheus-to-sd-exporter   k8s.gcr.io/metadata-proxy:v0.1.12,gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1                                           k8s-app=metadata-proxy,version=v0.1
      kube-system   daemonset.apps/node-local-dns                 3         3         3       3            3           addon.gke.io/node-local-dns-ds-ready=true                            5d22h   node-cache                                 gke.gcr.io/k8s-dns-node-cache:1.22.2-gke.0                                                                            k8s-app=node-local-dns
      kube-system   daemonset.apps/nvidia-gpu-device-plugin       0         0         0       0            0           <none>                                                               621d    nvidia-gpu-device-plugin                   gcr.io/gke-release/nvidia-gpu-device-plugin@sha256:80a976aecbc2c4cd4bae934807474bcd6f36baed94d4c122a52f87c04152e060   k8s-app=nvidia-gpu-device-plugin
      kube-system   daemonset.apps/pdcsi-node                     3         3         3       3            3           kubernetes.io/os=linux                                               598d    csi-driver-registrar,gce-pd-driver         gke.gcr.io/csi-node-driver-registrar:v2.5.0-gke.1,gke.gcr.io/gcp-compute-persistent-disk-csi-driver:v1.7.3-gke.1      k8s-app=gcp-compute-persistent-disk-csi-driver
      kube-system   daemonset.apps/pdcsi-node-windows             0         0         0       0            0           kubernetes.io/os=windows                                             535d    csi-driver-registrar,gce-pd-driver         gke.gcr.io/csi-node-driver-registrar:v2.5.0-gke.1,gke.gcr.io/gcp-compute-persistent-disk-csi-driver:v1.7.3-gke.1      k8s-app=gcp-compute-persistent-disk-csi-driver
      
      NAMESPACE        NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE      CONTAINERS                                 IMAGES                                                                                                                                                                     SELECTOR
      cert-manager     deployment.apps/cert-manager                         1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-controller:v1.9.1                                                                                                                            app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
      cert-manager     deployment.apps/cert-manager-cainjector              1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-cainjector:v1.9.1                                                                                                                            app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector
      cert-manager     deployment.apps/cert-manager-webhook                 1/1     1            1           98d      cert-manager                               quay.io/jetstack/cert-manager-webhook:v1.9.1                                                                                                                               app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
      custom-metrics   deployment.apps/custom-metrics-stackdriver-adapter   1/1     1            1           97d      pod-custom-metrics-stackdriver-adapter     gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0                                                                                                        k8s-app=custom-metrics-stackdriver-adapter,run=custom-metrics-stackdriver-adapter
      default          deployment.apps/****-api                   4/4     4            4           2y282d   ****-api                         gcr.io/****/api/main@sha256:a41a8cc6d249188c9145f43e4432ad0f966752d164fa8b637d1782c1724f739a                                                                     app=****-api
      default          deployment.apps/****-extension-api         4/4     4            4           2y283d   ****-extension-api               gcr.io/****/extension-api/main@sha256:4907584f47f2e1290d2e47027005b52759cdfa253d43ae4db06d1e94444ce5ab                                                           app=****-extension-api
      default          deployment.apps/****-extension-workers     6/6     6            6           2y38d    ****-extension-workers           gcr.io/****/extension-workers/main@sha256:5605c7334f73e6ff87f116fe39ef7da1a2052f9cbbab6225664688b99db1caed                                                       app=****-extension-workers
      default          deployment.apps/****-renderer              1/1     1            1           368d     ****-renderer                    gcr.io/****/renderer/main@sha256:2d72651a9393d8f5a701d18968b8cd88d55c385fd926c83922e090bcbf4f87cf                                                                app=****-renderer
      default          deployment.apps/****-webhook-api           2/2     2            2           79d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:eec95ede41e11cdb21309f085874d8fc2f273580d1457ac2d63d4c61cad35229                                                             app=****-webhook-api
      ingress-nginx    deployment.apps/ingress-nginx-controller             1/1     1            1           25m      controller                                 registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143                                                    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      kube-system      deployment.apps/event-exporter-gke                   1/1     1            1           2y23d    event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.9-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                           k8s-app=event-exporter
      kube-system      deployment.apps/konnectivity-agent                   3/3     3            3           362d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.31-gke.0                                                                                                                                       k8s-app=konnectivity-agent
      kube-system      deployment.apps/konnectivity-agent-autoscaler        1/1     1            1           362d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                     k8s-app=konnectivity-agent-autoscaler
      kube-system      deployment.apps/kube-dns                             2/2     2            2           3y58d    kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.3-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0   k8s-app=kube-dns
      kube-system      deployment.apps/kube-dns-autoscaler                  1/1     1            1           3y58d    autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                     k8s-app=kube-dns-autoscaler
      kube-system      deployment.apps/metrics-server-v0.4.5                1/1     1            1           236d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                         k8s-app=metrics-server,version=v0.4.5
      
      NAMESPACE        NAME                                                            DESIRED   CURRENT   READY   AGE      CONTAINERS                                 IMAGES                                                                                                                                                                                   SELECTOR
      cert-manager     replicaset.apps/cert-manager-7b8d75c477                         1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-controller:v1.9.1                                                                                                                                          app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=7b8d75c477
      cert-manager     replicaset.apps/cert-manager-cainjector-6cd8d7f84b              1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-cainjector:v1.9.1                                                                                                                                          app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=6cd8d7f84b
      cert-manager     replicaset.apps/cert-manager-webhook-64d76db6c                  1         1         1       98d      cert-manager                               quay.io/jetstack/cert-manager-webhook:v1.9.1                                                                                                                                             app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=64d76db6c
      custom-metrics   replicaset.apps/custom-metrics-stackdriver-adapter-848d58c7c7   1         1         1       97d      pod-custom-metrics-stackdriver-adapter     gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0                                                                                                                      k8s-app=custom-metrics-stackdriver-adapter,pod-template-hash=848d58c7c7,run=custom-metrics-stackdriver-adapter
      default          replicaset.apps/****-api-5488fb47c6                   0         0         0       15d      ****-api                         gcr.io/****/api/main@sha256:1439a994421ab2103ead8f8b7ac46d94eea8b2780a67eef7f07a7f3bf1a04ab3                                                                                   app=****-api,pod-template-hash=5488fb47c6
      default          replicaset.apps/****-api-586567774                    0         0         0       24d      ****-api                         gcr.io/****/api/main@sha256:dec6793c14b20db901826722befe7dff97a04099d275381098273d3b1bc7a64e                                                                                   app=****-api,pod-template-hash=586567774
      default          replicaset.apps/****-api-5876bb65                     0         0         0       4d22h    ****-api                         gcr.io/****/api/main@sha256:7c8cb3bb4e0ae7315ee5efb0676e7a9d3b594d04b6776cc443072f0af9aa76d0                                                                                   app=****-api,pod-template-hash=5876bb65
      default          replicaset.apps/****-api-69d6569587                   0         0         0       35d      ****-api                         gcr.io/****/api/main@sha256:b5f501548efc2eda05044db36cb6c93ca117f5a85962b13db1f2b09f12fe1a30                                                                                   app=****-api,pod-template-hash=69d6569587
      default          replicaset.apps/****-api-6d7b645f44                   0         0         0       39d      ****-api                         gcr.io/****/api/main@sha256:d6954db7435f091b52a06f39b0886399fecdb43e3bc7c8551d6fe067d3dcb22f                                                                                   app=****-api,pod-template-hash=6d7b645f44
      default          replicaset.apps/****-api-6fcfff547                    0         0         0       20d      ****-api                         gcr.io/****/api/main@sha256:b4a0534cf5a502eabba1aa61370a452068cf0704d0f4c3ff9b7a242cff3aab89                                                                                   app=****-api,pod-template-hash=6fcfff547
      default          replicaset.apps/****-api-77b785ff59                   0         0         0       35d      ****-api                         gcr.io/****/api/main@sha256:d65ab2c2694c18ed5806506b7a76cb641fc04071d0362cecec6a4e451c531a6f                                                                                   app=****-api,pod-template-hash=77b785ff59
      default          replicaset.apps/****-api-7b5c9dd98b                   4         4         4       3d22h    ****-api                         gcr.io/****/api/main@sha256:a41a8cc6d249188c9145f43e4432ad0f966752d164fa8b637d1782c1724f739a                                                                                   app=****-api,pod-template-hash=7b5c9dd98b
      default          replicaset.apps/****-api-7f987f4b84                   0         0         0       27d      ****-api                         gcr.io/****/api/main@sha256:3f35a92f97d417cefb54ab7221442b3c5ff64339f32c8f32fb3dd3b2feb69b9b                                                                                   app=****-api,pod-template-hash=7f987f4b84
      default          replicaset.apps/****-api-df967849c                    0         0         0       23d      ****-api                         gcr.io/****/api/main@sha256:b1e53c55254b319d6f0232106a444570d8dab0facddb214dc84e185d79d46cc8                                                                                   app=****-api,pod-template-hash=df967849c
      default          replicaset.apps/****-api-fb656fc69                    0         0         0       24d      ****-api                         gcr.io/****/api/main@sha256:01544143593b8096fc3c4a61de210ba01b7a4e2de8bd5c9fef69c8dad896bcbf                                                                                   app=****-api,pod-template-hash=fb656fc69
      default          replicaset.apps/****-extension-api-5685d8c768         0         0         0       72d      ****-extension-api               gcr.io/****/extension-api/main@sha256:b7514236bcc5330e053a3d08df3ebb3e43b2153f8ed23fe1f57cf9c87bc2fe13                                                                         app=****-extension-api,pod-template-hash=5685d8c768
      default          replicaset.apps/****-extension-api-59dcd6d7cc         0         0         0       35d      ****-extension-api               gcr.io/****/extension-api/main@sha256:922d644fd379eaafa64c82a2b8c1b81efc49f896c96a2b8497f4cb59a728c0c4                                                                         app=****-extension-api,pod-template-hash=59dcd6d7cc
      default          replicaset.apps/****-extension-api-5fd9dd57d6         0         0         0       35d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1dd2368f94dce47c9acea2756f0b862e66ca116d9bedde0a10542911d8b05210                                                                         app=****-extension-api,pod-template-hash=5fd9dd57d6
      default          replicaset.apps/****-extension-api-64964c8b7          4         4         4       15d      ****-extension-api               gcr.io/****/extension-api/main@sha256:4907584f47f2e1290d2e47027005b52759cdfa253d43ae4db06d1e94444ce5ab                                                                         app=****-extension-api,pod-template-hash=64964c8b7
      default          replicaset.apps/****-extension-api-654db46b5f         0         0         0       70d      ****-extension-api               gcr.io/****/extension-api/main@sha256:3703ef5cacbb442cc9f65a7f028bf21a28a082b38a938892b5ca0abe15e40ecc                                                                         app=****-extension-api,pod-template-hash=654db46b5f
      default          replicaset.apps/****-extension-api-6559bb9d55         0         0         0       56d      ****-extension-api               gcr.io/****/extension-api/main@sha256:7cb4b299427172e3c0f3da6d908a5e7c64a338c9773e9696357b2b8a23d040ad                                                                         app=****-extension-api,pod-template-hash=6559bb9d55
      default          replicaset.apps/****-extension-api-676dcdcdf4         0         0         0       77d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1f807cf9d0aa5f8eaa6a078b6bcb0c4a2474641e18942b4f6de0f2f240e4454e                                                                         app=****-extension-api,pod-template-hash=676dcdcdf4
      default          replicaset.apps/****-extension-api-74764fc5b9         0         0         0       24d      ****-extension-api               gcr.io/****/extension-api/main@sha256:26c01c24cf2a9a156d12b40df0106174e6f04c482bf83d2314c734d604bb927d                                                                         app=****-extension-api,pod-template-hash=74764fc5b9
      default          replicaset.apps/****-extension-api-757d8bdc5c         0         0         0       24d      ****-extension-api               gcr.io/****/extension-api/main@sha256:1e647caa6569e6a7a047be50b2cd3c40a18e9fd71ca334c8b8c0460b83516244                                                                         app=****-extension-api,pod-template-hash=757d8bdc5c
      default          replicaset.apps/****-extension-api-85d9dd68bd         0         0         0       23d      ****-extension-api               gcr.io/****/extension-api/main@sha256:8a4c00850c8719f4f6cf5a33fd400898de91b3fda421220a374aa1663e928fb4                                                                         app=****-extension-api,pod-template-hash=85d9dd68bd
      default          replicaset.apps/****-extension-api-cf6cb5d5c          0         0         0       27d      ****-extension-api               gcr.io/****/extension-api/main@sha256:62fb63417fdec37d77f899db4e49f53ff09cc0a4d7698d3a490713f2b923fcc8                                                                         app=****-extension-api,pod-template-hash=cf6cb5d5c
      default          replicaset.apps/****-extension-workers-548cb46fdc     0         0         0       23d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:d1e8ca29ca2211638fa75c722e83691f4d0414ce1ccbfc6e2ddc57135ea26c1b                                                                     app=****-extension-workers,pod-template-hash=548cb46fdc
      default          replicaset.apps/****-extension-workers-5b66577498     0         0         0       21d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:783ab87a076616c3da7db42f3847dbd46354495c046213736a1a84ca348df7d5                                                                     app=****-extension-workers,pod-template-hash=5b66577498
      default          replicaset.apps/****-extension-workers-5c847c5456     6         6         6       15d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:5605c7334f73e6ff87f116fe39ef7da1a2052f9cbbab6225664688b99db1caed                                                                     app=****-extension-workers,pod-template-hash=5c847c5456
      default          replicaset.apps/****-extension-workers-664b88db8c     0         0         0       25d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:4b3af3c696cf9e539f2697f0c338f804aa8019177000143302e38fb2bf5c574c                                                                     app=****-extension-workers,pod-template-hash=664b88db8c
      default          replicaset.apps/****-extension-workers-68cbd968cc     0         0         0       18d      ****-extension-workers           gcr.io/****/extension-workers/pull/554@sha256:3b17e84bb29fad14e009496288de8954caa314f8fc0874d925eae6c2616b3af3                                                                 app=****-extension-workers,pod-template-hash=68cbd968cc
      default          replicaset.apps/****-extension-workers-6f5854878      0         0         0       25d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:8f752bf2fa4ecb3d040d9a345a1985b554b05c3b2fe570ebc2e818f3cfb68059                                                                     app=****-extension-workers,pod-template-hash=6f5854878
      default          replicaset.apps/****-extension-workers-76c48dd85c     0         0         0       23d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:2d2bcdd4ebac4d69607e32372b95f2994ba46ad1be49c2396e2a878e73bb05ea                                                                     app=****-extension-workers,pod-template-hash=76c48dd85c
      default          replicaset.apps/****-extension-workers-76d9d5b69c     0         0         0       27d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:b933924ef2ab0c36c640d15d0449c7cf63e147a6cb0f68f170e4e74a176d3e06                                                                     app=****-extension-workers,pod-template-hash=76d9d5b69c
      default          replicaset.apps/****-extension-workers-799fd4b667     0         0         0       20d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:8d2327560cdc3d7011d8b53e52dbd364ba23b179c0f0b48df4d53db5c68decef                                                                     app=****-extension-workers,pod-template-hash=799fd4b667
      default          replicaset.apps/****-extension-workers-7d7bc97d65     0         0         0       24d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:7421e698772b4c740f792b41ca1ad82a01d4a87f608330a42ba5a4ebd5ee31c9                                                                     app=****-extension-workers,pod-template-hash=7d7bc97d65
      default          replicaset.apps/****-extension-workers-8647d57898     0         0         0       24d      ****-extension-workers           gcr.io/****/extension-workers/main@sha256:88275fac722a7d42314dcd9f0a757e7c0db461855b3b56f2226400300120b7e9                                                                     app=****-extension-workers,pod-template-hash=8647d57898
      default          replicaset.apps/****-renderer-54cd644656              0         0         0       56d      ****-renderer                    gcr.io/****/renderer/main@sha256:07a6585e5238e2d63df117c18ecd9b8d5a57fbfa9765b6babb0707d56b845246                                                                              app=****-renderer,pod-template-hash=54cd644656
      default          replicaset.apps/****-renderer-5589dc84d9              0         0         0       35d      ****-renderer                    gcr.io/****/renderer/main@sha256:90fc79630921b7a67b86df450c5ae386994e3b2276bd911c83b8c6e4bd1943b4                                                                              app=****-renderer,pod-template-hash=5589dc84d9
      default          replicaset.apps/****-renderer-58db5ddd8d              0         0         0       24d      ****-renderer                    gcr.io/****/renderer/main@sha256:bfed03c7bfc07153a4c7f382de61f67ada23647a25d32e0122689aa9019f549a                                                                              app=****-renderer,pod-template-hash=58db5ddd8d
      default          replicaset.apps/****-renderer-5d4464b756              0         0         0       77d      ****-renderer                    gcr.io/****/renderer/main@sha256:228364867a0a645b64bcd10fcb2c2c1c5f060928eb2c7fb0989f718d879fbcf4                                                                              app=****-renderer,pod-template-hash=5d4464b756
      default          replicaset.apps/****-renderer-5ff77dd894              0         0         0       24d      ****-renderer                    gcr.io/****/renderer/main@sha256:4004c35f60c0efe80a58aeafbe534a5bd77ec8c261c96fec167a509d2ae31fe8                                                                              app=****-renderer,pod-template-hash=5ff77dd894
      default          replicaset.apps/****-renderer-6554bfdf4               0         0         0       35d      ****-renderer                    gcr.io/****/renderer/main@sha256:a9b42e5d92092ece7b120406ca07aa01efef5c11461baf4eaf25182b4afd2c61                                                                              app=****-renderer,pod-template-hash=6554bfdf4
      default          replicaset.apps/****-renderer-66959b6dc5              1         1         1       15d      ****-renderer                    gcr.io/****/renderer/main@sha256:2d72651a9393d8f5a701d18968b8cd88d55c385fd926c83922e090bcbf4f87cf                                                                              app=****-renderer,pod-template-hash=66959b6dc5
      default          replicaset.apps/****-renderer-76b847c999              0         0         0       27d      ****-renderer                    gcr.io/****/renderer/main@sha256:e8d2916a5929dfeee8748fe77b357d658b80408712b1754123be24f8d778b01f                                                                              app=****-renderer,pod-template-hash=76b847c999
      default          replicaset.apps/****-renderer-7b57bd4d98              0         0         0       72d      ****-renderer                    gcr.io/****/renderer/main@sha256:b83b0f24397d8e0127f8dc3616fe257bd2178acd3d841eb02a5d98442dbdc334                                                                              app=****-renderer,pod-template-hash=7b57bd4d98
      default          replicaset.apps/****-renderer-85866fbc7c              0         0         0       70d      ****-renderer                    gcr.io/****/renderer/main@sha256:faab61a914cefd4d24537a43886eced978b4cf6bd486425a4a1cf1ee02d2e865                                                                              app=****-renderer,pod-template-hash=85866fbc7c
      default          replicaset.apps/****-renderer-d58fb68b5               0         0         0       23d      ****-renderer                    gcr.io/****/renderer/main@sha256:2d80b6c3b8cee7ba3f45eab53b83f75954c068b732ba07f39cd681129a6c3971                                                                              app=****-renderer,pod-template-hash=d58fb68b5
      default          replicaset.apps/****-webhook-api-58946cd95b           0         0         0       35d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:b2a3965ed89f284cc54fc7b6b51e2d679ad2889649c992d01794dd4dbc2e8e1a                                                                           app=****-webhook-api,pod-template-hash=58946cd95b
      default          replicaset.apps/****-webhook-api-5fb64b4958           0         0         0       23d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:4c699a28570bf689d512361be8f1260a43c570d062a1491f2d00116094ad85fa                                                                           app=****-webhook-api,pod-template-hash=5fb64b4958
      default          replicaset.apps/****-webhook-api-5fcfd5d96b           0         0         0       70d      ****-webhook-api                 gcr.io/****/webhook-api/pull/527@sha256:bc1971939772069843611afbcb389d7d27e793d25d8311860f973611640099aa                                                                       app=****-webhook-api,pod-template-hash=5fcfd5d96b
      default          replicaset.apps/****-webhook-api-686475565c           0         0         0       56d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:001c1293692dc38c810d2440ce5a4bdf99db65339724250d8f73b265aa8aa958                                                                           app=****-webhook-api,pod-template-hash=686475565c
      default          replicaset.apps/****-webhook-api-6f7bdf6d99           0         0         0       24d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:9882451f7a39ad9ec5d0a31fb3f7f4bef34666bab58e0305981e5100ba44852f                                                                           app=****-webhook-api,pod-template-hash=6f7bdf6d99
      default          replicaset.apps/****-webhook-api-6f8db6fb             0         0         0       24d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:67fcfb97e04fd26e77ebf8e48310a29e0441d9ea7ed04524e41d27cdb80ed2f9                                                                           app=****-webhook-api,pod-template-hash=6f8db6fb
      default          replicaset.apps/****-webhook-api-6fcd976b57           0         0         0       27d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:11390b9ac2b2800c522fc89afacee26798b6e022028a0506e18a004e55fc33f1                                                                           app=****-webhook-api,pod-template-hash=6fcd976b57
      default          replicaset.apps/****-webhook-api-7776866645           0         0         0       52d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:b2a6a4d81a6eae7b728e7fb0a217f3196684dc74caefb04bef3c9ffa1cffe5e7                                                                           app=****-webhook-api,pod-template-hash=7776866645
      default          replicaset.apps/****-webhook-api-86bb7d96fd           2         2         2       15d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:eec95ede41e11cdb21309f085874d8fc2f273580d1457ac2d63d4c61cad35229                                                                           app=****-webhook-api,pod-template-hash=86bb7d96fd
      default          replicaset.apps/****-webhook-api-979cd4fc9            0         0         0       70d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:646941a68e7691448e43cee142daebe071c14f62af8986c2f0ef427be5ba4797                                                                           app=****-webhook-api,pod-template-hash=979cd4fc9
      default          replicaset.apps/****-webhook-api-cc65bb7b6            0         0         0       35d      ****-webhook-api                 gcr.io/****/webhook-api/main@sha256:dade0020a9c87760588783094fa608b688d00149c122bc6e9f1fa5724f5df24b                                                                           app=****-webhook-api,pod-template-hash=cc65bb7b6
      ingress-nginx    replicaset.apps/ingress-nginx-controller-69fbfb4bfd             1         1         1       25m      controller                                 registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143                                                                  app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=69fbfb4bfd
      kube-system      replicaset.apps/event-exporter-gke-5479fd58c8                   0         0         0       236d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.5-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=5479fd58c8
      kube-system      replicaset.apps/event-exporter-gke-666b7ffbf7                   0         0         0       663d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=666b7ffbf7
      kube-system      replicaset.apps/event-exporter-gke-77cccd97c6                   0         0         0       696d     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=77cccd97c6
      kube-system      replicaset.apps/event-exporter-gke-8489df9489                   0         0         0       2y23d    event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=8489df9489
      kube-system      replicaset.apps/event-exporter-gke-f66d9f855                    1         1         1       3d5h     event-exporter,prometheus-to-sd-exporter   gke.gcr.io/event-exporter:v0.3.9-gke.0,gke.gcr.io/prometheus-to-sd:v0.10.0-gke.0                                                                                                         k8s-app=event-exporter,pod-template-hash=f66d9f855
      kube-system      replicaset.apps/konnectivity-agent-54dcfd5b75                   0         0         0       235d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=54dcfd5b75
      kube-system      replicaset.apps/konnectivity-agent-68dbdb55                     0         0         0       102d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.27-gke.2                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=68dbdb55
      kube-system      replicaset.apps/konnectivity-agent-6fb49c9c54                   0         0         0       166d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=6fb49c9c54
      kube-system      replicaset.apps/konnectivity-agent-7765499767                   0         0         0       236d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.24-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=7765499767
      kube-system      replicaset.apps/konnectivity-agent-7998ff9dd6                   0         0         0       362d     konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.22-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=7998ff9dd6
      kube-system      replicaset.apps/konnectivity-agent-8484759759                   3         3         3       38d      konnectivity-agent                         gke.gcr.io/proxy-agent:v0.0.31-gke.0                                                                                                                                                     k8s-app=konnectivity-agent,pod-template-hash=8484759759
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-5c49cb58bb        0         0         0       236d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=5c49cb58bb
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6b86f667c9        0         0         0       166d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6b86f667c9
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6cb774c9cc        0         0         0       362d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6cb774c9cc
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-6dfb4f9cfb        1         1         1       38d      autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=6dfb4f9cfb
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-7fd5dd4f5         0         0         0       102d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=7fd5dd4f5
      kube-system      replicaset.apps/konnectivity-agent-autoscaler-ddccb8b95         0         0         0       235d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=konnectivity-agent-autoscaler,pod-template-hash=ddccb8b95
      kube-system      replicaset.apps/kube-dns-56494768b7                             0         0         0       102d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.21.4-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.21.4-gke.0,gke.gcr.io/k8s-dns-sidecar:1.21.4-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=56494768b7
      kube-system      replicaset.apps/kube-dns-5d54b45645                             0         0         0       535d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.17.1-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.17.1-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.17.1-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2      k8s-app=kube-dns,pod-template-hash=5d54b45645
      kube-system      replicaset.apps/kube-dns-6465f78586                             0         0         0       572d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=6465f78586
      kube-system      replicaset.apps/kube-dns-66d6b7c877                             0         0         0       663d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.15-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.15-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.15-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=66d6b7c877
      kube-system      replicaset.apps/kube-dns-6bd88c9b66                             0         0         0       621d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=6bd88c9b66
      kube-system      replicaset.apps/kube-dns-6c7b8dc9f9                             0         0         0       506d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.17.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.17.3-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.17.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2      k8s-app=kube-dns,pod-template-hash=6c7b8dc9f9
      kube-system      replicaset.apps/kube-dns-7f4d6f474d                             0         0         0       270d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.21.0-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.21.0-gke.0,gke.gcr.io/k8s-dns-sidecar:1.21.0-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2                        k8s-app=kube-dns,pod-template-hash=7f4d6f474d
      kube-system      replicaset.apps/kube-dns-85df8994db                             0         0         0       94d      kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.2-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.2-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.2-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=85df8994db
      kube-system      replicaset.apps/kube-dns-9c59558bb                              0         0         0       642d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.16-gke.0,gke.gcr.io/k8s-dns-sidecar-amd64:1.15.16-gke.0,gke.gcr.io/prometheus-to-sd:v0.4.2   k8s-app=kube-dns,pod-template-hash=9c59558bb
      kube-system      replicaset.apps/kube-dns-autoscaler-58cbd4f75c                  0         0         0       535d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=58cbd4f75c
      kube-system      replicaset.apps/kube-dns-autoscaler-5c78d65cd9                  0         0         0       2y       autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=5c78d65cd9
      kube-system      replicaset.apps/kube-dns-autoscaler-645f7d66cf                  0         0         0       2y23d    autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=645f7d66cf
      kube-system      replicaset.apps/kube-dns-autoscaler-76fcd5f658                  0         0         0       3y58d    autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.2.0                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=76fcd5f658
      kube-system      replicaset.apps/kube-dns-autoscaler-7f89fb6b79                  0         0         0       621d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=7f89fb6b79
      kube-system      replicaset.apps/kube-dns-autoscaler-844c9d9448                  0         0         0       460d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=844c9d9448
      kube-system      replicaset.apps/kube-dns-autoscaler-85f8bdb54                   0         0         0       2y314d   autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=85f8bdb54
      kube-system      replicaset.apps/kube-dns-autoscaler-8687c64fc                   0         0         0       2y238d   autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler-amd64:1.7.1-gke.0                                                                                                                             k8s-app=kube-dns-autoscaler,pod-template-hash=8687c64fc
      kube-system      replicaset.apps/kube-dns-autoscaler-bb58c6784                   0         0         0       2y314d   autoscaler                                 k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.3.0                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=bb58c6784
      kube-system      replicaset.apps/kube-dns-autoscaler-f4d55555                    1         1         1       102d     autoscaler                                 gke.gcr.io/cluster-proportional-autoscaler:1.8.4-gke.1                                                                                                                                   k8s-app=kube-dns-autoscaler,pod-template-hash=f4d55555
      kube-system      replicaset.apps/kube-dns-b4f5c58c7                              0         0         0       418d     kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.19.1-gke.1,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.19.1-gke.1,gke.gcr.io/k8s-dns-sidecar:1.19.1-gke.1,gke.gcr.io/prometheus-to-sd:v0.4.2                        k8s-app=kube-dns,pod-template-hash=b4f5c58c7
      kube-system      replicaset.apps/kube-dns-b99c7865d                              2         2         2       44d      kubedns,dnsmasq,sidecar,prometheus-to-sd   gke.gcr.io/k8s-dns-kube-dns:1.22.3-gke.0,gke.gcr.io/k8s-dns-dnsmasq-nanny:1.22.3-gke.0,gke.gcr.io/k8s-dns-sidecar:1.22.3-gke.0,gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0                 k8s-app=kube-dns,pod-template-hash=b99c7865d
      kube-system      replicaset.apps/metrics-server-v0.4.5-56fcb96597                0         0         0       236d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=56fcb96597,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-57bfdff4dd                0         0         0       128d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=57bfdff4dd,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-5bdb996c9f                0         0         0       232d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=5bdb996c9f,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-68f97d654f                0         0         0       153d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=68f97d654f,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-788d9b568b                0         0         0       88d      metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=788d9b568b,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-bbb794dcc                 0         0         0       236d     metrics-server,metrics-server-nanny        gcr.io/gke-release/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                               k8s-app=metrics-server,pod-template-hash=bbb794dcc,version=v0.4.5
      kube-system      replicaset.apps/metrics-server-v0.4.5-fb4c49dd6                 1         1         1       126d     metrics-server,metrics-server-nanny        gke.gcr.io/metrics-server-amd64:v0.4.5-gke.0,gke.gcr.io/addon-resizer:1.8.13-gke.0                                                                                                       k8s-app=metrics-server,pod-template-hash=fb4c49dd6,version=v0.4.5
      
      NAMESPACE   NAME                          READY   AGE    CONTAINERS       IMAGES
      default     statefulset.apps/mongodb      3/3     591d   mongodb          docker.io/bitnami/mongodb:4.4.4-debian-10-r30
      default     statefulset.apps/redis-node   3/3     433d   redis,sentinel   docker.io/bitnami/redis:6.2.5-debian-10-r34,docker.io/bitnami/redis-sentinel:6.2.5-debian-10-r34
      
      NAMESPACE   NAME                                                                   REFERENCE                                     TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
      default     horizontalpodautoscaler.autoscaling/****-extension-workers   Deployment/****-extension-workers   80666m/75   4         40        6          98d
      
    • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
      Name:                     ingress-nginx-controller
      Namespace:                ingress-nginx
      Labels:                   app.kubernetes.io/component=controller
                                app.kubernetes.io/instance=ingress-nginx
                                app.kubernetes.io/managed-by=Helm
                                app.kubernetes.io/name=ingress-nginx
                                app.kubernetes.io/part-of=ingress-nginx
                                app.kubernetes.io/version=1.4.0
                                helm.sh/chart=ingress-nginx-4.3.0
      Annotations:              meta.helm.sh/release-name: ingress-nginx
                                meta.helm.sh/release-namespace: ingress-nginx
      Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      Type:                     LoadBalancer
      IP Family Policy:         SingleStack
      IP Families:              IPv4
      IP:                       10.20.15.191
      IPs:                      10.20.15.191
      LoadBalancer Ingress:     35.193.145.228
      Port:                     http  80/TCP
      TargetPort:               http/TCP
      NodePort:                 http  32447/TCP
      Endpoints:                10.16.1.25:80
      Port:                     https  443/TCP
      TargetPort:               https/TCP
      NodePort:                 https  30526/TCP
      Endpoints:                10.16.1.25:443
      Session Affinity:         None
      External Traffic Policy:  Local
      HealthCheck NodePort:     31493
      Events:
        Type    Reason                Age   From                Message
        ----    ------                ----  ----                -------
        Normal  EnsuringLoadBalancer  26m   service-controller  Ensuring load balancer
        Normal  EnsuredLoadBalancer   25m   service-controller  Ensured load balancer
      
      Name:              ingress-nginx-controller-admission
      Namespace:         ingress-nginx
      Labels:            app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.4.0
                        helm.sh/chart=ingress-nginx-4.3.0
      Annotations:       meta.helm.sh/release-name: ingress-nginx
                        meta.helm.sh/release-namespace: ingress-nginx
      Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      Type:              ClusterIP
      IP Family Policy:  SingleStack
      IP Families:       IPv4
      IP:                10.20.6.148
      IPs:               10.20.6.148
      Port:              https-webhook  443/TCP
      TargetPort:        webhook/TCP
      Endpoints:         10.16.1.25:8443
      Session Affinity:  None
      Events:            <none>
      
  • Current state of ingress object, if applicable:

    • kubectl -n <appnnamespace> get all,ing -o wide
      NAME                                            READY   STATUS    RESTARTS   AGE   IP           NODE                                                  NOMINATED NODE   READINESS GATES
      pod/ingress-nginx-controller-69fbfb4bfd-k29tz   1/1     Running   0          32m   10.16.1.25   gke-overlay-expert-preemptive-pool-e2-e219fbd6-54gl   <none>           <none>
      
      NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE   SELECTOR
      service/ingress-nginx-controller             LoadBalancer   10.20.15.191   35.193.145.228   80:32447/TCP,443:30526/TCP   32m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      service/ingress-nginx-controller-admission   ClusterIP      10.20.6.148    <none>           443/TCP                      32m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      
      NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
      deployment.apps/ingress-nginx-controller   1/1     1            1           32m   controller   registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
      
      NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
      replicaset.apps/ingress-nginx-controller-69fbfb4bfd   1         1         1       32m   controller   registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=69fbfb4bfd
      
    • kubectl -n <appnamespace> describe ing <ingressname>: n/a
    • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag: n/a

Anything else we need to know:

I have the following ExternalName service:

kind: Service
apiVersion: v1
metadata:
  name: ****-storage
  namespace: default
spec:
  type: ExternalName
  externalName: storage.googleapis.com

I followed this comment and it resolved #6060, but now I am experiencing this issue.

This issue might be related to #3882, #4551, #6523 and #8941.

I also realize that I am on an older version. I have been hesitant/scared to upgrade 😅 but will upgrading resolve the issue?

Any help will be greatly appreciated.

@marcandrews marcandrews added the kind/bug Categorizes issue or PR as related to a bug. label Nov 2, 2022
@k8s-ci-robot
Copy link
Contributor

@marcandrews: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Nov 2, 2022
@longwuyuan
Copy link
Contributor

longwuyuan commented Nov 2, 2022 via email

@marcandrews
Copy link
Author

I figured as much 😄

I will upgrade this weekend and report back.

@longwuyuan
Copy link
Contributor

longwuyuan commented Nov 2, 2022 via email

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Nov 2, 2022
@marcandrews
Copy link
Author

@longwuyuan I upgraded to v1.4.0 and I am still experiencing the issue.

I noticed that nothing in my cluster is associated 10.16.0.23 or 10.16.4.24 so I'm unsure why Nginx would be looking for those IPs. Maybe there's some old config somewhere? Any tips to help me debug?

@marcandrews
Copy link
Author

marcandrews commented Nov 5, 2022

OK I think I figured it out.

I ran kubectl ingress-nginx backends and it showed the following backend:

[
  {
    "name": "custom-default-backend-default-****-api",
    "service": {
      "metadata": {
        "creationTimestamp": null
      },
      "spec": {
        "type": "ExternalName",
        "sessionAffinity": "None",
        "externalName": "storage.googleapis.com",
        "internalTrafficPolicy": "Cluster"
      },
      "status": {
        "loadBalancer": {}
      }
    },
    "port": 80,
    "sslPassthrough": false,
    "endpoints": [
      {
        "address": "10.16.0.23",
        "port": "6000"
      },
      {
        "address": "10.16.4.23",
        "port": "6000"
      },
      {
        "address": "10.16.0.24",
        "port": "6000"
      },
      {
        "address": "10.16.4.24",
        "port": "6000"
      }
    ],
    "sessionAffinityConfig": {
      "name": "",
      "mode": "",
      "cookieSessionAffinity": {
        "name": ""
      }
    },
    "upstreamHashByConfig": {
      "upstream-hash-by-subset-size": 3
    },
    "noServer": false,
    "trafficShapingPolicy": {
      "weight": 0,
      "weightTotal": 0,
      "header": "",
      "headerValue": "",
      "headerPattern": "",
      "cookie": ""
    }
  },
  ...
]

Then I looked at my ingresses and found that one of them had:

metadata:
  annotations:
    ...
    nginx.ingress.kubernetes.io/default-backend: overlay-expert-api

Once I removed that annotation, that backend no longer registered and the error no longer appears.

The ingress in question is for the ExternalName service shown in the OP and I believe what I was trying to accomplish was to forward 404 originating from this ExternalName service to another service. After removing this annotation, the native Nginx 404 page is used. What's the best way to fallback to another service? (edit follow this maybe?)

@marcandrews
Copy link
Author

Yep, this is how to fallback to another service. Thanks @longwuyuan ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Development

No branches or pull requests

3 participants