tigera-operator docs for EKS contains deprecated API versions #6491

barsilver · 2022-08-03T09:02:04Z

I'm upgrading EKS from 1.21 to the 1.22 version. PodSecurityPolicy in the policy/v1beta1 API version is being deprecated in Kubernetes 1.21 and will no longer be served in v1.25. I followed the documentation here for the operator installation but the instructions aren't updated for newer Kubernetes versions which cannot deploy PodSecurityPolicy (1.22+).

Expected Behavior

Installing Calico using tigera-operator.yaml from the documentation should be compatible with newer Kubernetes versions as it is for the calico installation from manifest

Current Behavior

tigera-operator installation as mentioned in the EKS documentation of project calico installs old API versions which are deprecated in k8s v1.22+

Your Environment

Calico version: v3.23.1
Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.21

barsilver · 2022-08-03T09:28:06Z

I tried to run kubectl apply for this tigera-operator.yaml but it failed with the following error:

The CustomResourceDefinition "installations.operator.tigera.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

barsilver · 2022-08-03T09:36:53Z

kubectl replace for this YAML file was the solution

barsilver · 2022-08-03T10:53:33Z

Apparently, it wasn't the solution, because after it finished progressing and rebooted all the pods in calico-system tigerastatus looked like this:

NAME     AVAILABLE   PROGRESSING   DEGRADED   SINCE
calico   True        False         False      6s

Few seconds after, the pods started to reboot again and that was the tigerastatus output:

Name:         calico
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  operator.tigera.io/v1
Kind:         TigeraStatus
Metadata:
  Creation Timestamp:  2022-06-23T09:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  operator.tigera.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
      f:status:
        .:
        f:conditions:
    Manager:         operator
    Operation:       Update
    Time:            2022-06-23T09:45:34Z
  Resource Version:  671188473
  UID:               b50f6c9c-1ffb-4ee2-a6ff-bf498d905453
Spec:
Status:
  Conditions:
    Last Transition Time:  2022-08-03T10:41:50Z
    Observed Generation:   5
    Reason:                Unknown
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2022-08-03T10:43:50Z
    Observed Generation:   5
    Reason:                Unknown
    Status:                False
    Type:                  Available
    Last Transition Time:  2022-08-03T10:43:50Z
    Message:               DaemonSet "calico-system/calico-node" update is rolling out (15 out of 60 updated)
    Observed Generation:   5
    Reason:                ResourceNotReady
    Status:                True
    Type:                  Progressing
Events:                    <none>

The logs from from the tigera-operator namespace:

tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523611.8609326,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523611.8609657,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523635.454624,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523635.4546576,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523642.0554605,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523642.0554945,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523672.235133,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523672.2351656,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523702.4124248,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523702.4124672,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523710.454353,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523710.454387,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523730.5845706,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523730.5846014,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523732.5866766,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523732.5867133,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523745.4569619,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523745.4569979,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523755.4577687,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523755.457799,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523762.768554,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523762.768602,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523775.4567487,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523775.4567802,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523792.9404783,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523792.940505,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523800.4550567,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523800.455081,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523815.454971,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523815.455004,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523823.119496,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523823.119522,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523835.4547129,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523835.4547586,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523845.4552584,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523845.4552994,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523853.303227,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523853.303257,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523860.4542513,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523860.4542983,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523875.451944,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523875.4519773,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523883.4882042,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523883.4882388,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523890.4543114,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523890.4543421,"logger":"controller_apiserver","msg":"APIServer config not found","Request.Namespace":"","Request.Name":"default"}
tigera-operator-7787548745-dcwp8 tigera-operator {"level":"info","ts":1659523905.453304,"logger":"controller_apiserver","msg":"Reconciling APIServer","Request.Namespace":"","Request.Name":"default"}

caseydavenport · 2022-08-03T18:59:35Z

Yes, we need to update the EKS documentation to use a newer version.

Did you install via helm originally?

barsilver · 2022-08-06T09:48:55Z

@caseydavenport
No, I installed it as described in the documentation with the following command:

kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml

But in order to upgrade it, since I didn't use helm in the first place, I used the command:

kubectl replace -f https://github.com/raw/projectcalico/calico/master/manifests/tigera-operator.yaml

EDIT: Installing tigera-operator via helm created the apiserver in tigerastatus and solved the errors of "APIServer config not found", but PodSecurityPolicy resources are still being deployed.

caseydavenport · 2022-08-08T17:10:37Z

PodSecurityPolicy has been removed in master and the upcoming v3.24 release:

Once v3.24 is out, we just need to update the EKS documentation.

lwr20 · 2022-08-23T16:49:18Z

tigera-operator installation as mentioned in the EKS documentation of project calico installs old API versions which are deprecated in k8s v1.22+

This is deliberate. By continuing to use deprecated k8s APIs, we can ensure that the compatibility of Calico with multiple versions of k8s. (arguably this is the point of deprecation)

We typically switch to the new API just before the removal of the API from k8s - e.g. in this case Calico v3.24 was the last Calico release before k8s 1.25 came out and removed this API, so that is the release we switched from PodSecurityPolicies to PodSecurityStandards.

caseydavenport · 2022-08-26T17:39:17Z

v3.24 is out now, so we're clear to update the docs. I believe @coutinhop was looking at making some changes in that area already?

viceice · 2022-09-21T11:06:17Z

I still get The CustomResourceDefinition "installations.operator.tigera.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes error 😕

caseydavenport · 2022-09-21T16:45:43Z

@viceice use kubectl create and kubectl replace for CRDs, rather than using kubectl apply - that should avoid that error.

caseydavenport · 2022-09-21T16:46:41Z

kubectl puts annotations on objects when using apply, and the annotation it creates in this case it not valid.

We need to figure out if there's a way we can avoid this situation, but in general kubectl create and replace are better options regardless.

shu-mutou · 2022-09-27T09:18:44Z

@caseydavenport Thanks! kubectl create instead of apply worked fine for my local cluster.

coutinhop · 2023-03-06T17:46:15Z

Closing this as using kubectl create and kubectl replace for CRDs look like good solutions. Feel free to reopen if there are still issues.

medoedoff · 2023-11-03T13:43:56Z

Still issue but in ArgoCD. I created application tigera-operator and faced with the same issue.

ArgoCD application manifset:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tigera-operator
  namespace: argocd
spec:
  project: third-party
  source:
    repoURL: 'https://docs.projectcalico.org/charts/'
    targetRevision: 3.26.3
    chart: tigera-operator
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: tigera-operator
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - PruneLast=true
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

lwr20 · 2023-11-03T14:39:39Z

@medoedoff see
https://foxutech.medium.com/how-to-fix-too-long-must-have-at-most-262144-bytes-in-argocd-2a00cddbbe99

barsilver closed this as completed Aug 3, 2022

barsilver reopened this Aug 3, 2022

coutinhop added the kind/support label Aug 9, 2022

sp3nx0r mentioned this issue Sep 24, 2022

Upgrading Calico onedr0p/cluster-template#321

Closed

coutinhop closed this as completed Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tigera-operator docs for EKS contains deprecated API versions #6491

tigera-operator docs for EKS contains deprecated API versions #6491

barsilver commented Aug 3, 2022 •

edited

Loading

barsilver commented Aug 3, 2022

barsilver commented Aug 3, 2022

barsilver commented Aug 3, 2022

caseydavenport commented Aug 3, 2022

barsilver commented Aug 6, 2022 •

edited

Loading

caseydavenport commented Aug 8, 2022

lwr20 commented Aug 23, 2022 •

edited

Loading

caseydavenport commented Aug 26, 2022

viceice commented Sep 21, 2022

caseydavenport commented Sep 21, 2022

caseydavenport commented Sep 21, 2022

shu-mutou commented Sep 27, 2022

coutinhop commented Mar 6, 2023

medoedoff commented Nov 3, 2023

lwr20 commented Nov 3, 2023 •

edited

Loading

tigera-operator docs for EKS contains deprecated API versions #6491

tigera-operator docs for EKS contains deprecated API versions #6491

Comments

barsilver commented Aug 3, 2022 • edited Loading

Expected Behavior

Current Behavior

Your Environment

barsilver commented Aug 3, 2022

barsilver commented Aug 3, 2022

barsilver commented Aug 3, 2022

caseydavenport commented Aug 3, 2022

barsilver commented Aug 6, 2022 • edited Loading

caseydavenport commented Aug 8, 2022

lwr20 commented Aug 23, 2022 • edited Loading

caseydavenport commented Aug 26, 2022

viceice commented Sep 21, 2022

caseydavenport commented Sep 21, 2022

caseydavenport commented Sep 21, 2022

shu-mutou commented Sep 27, 2022

coutinhop commented Mar 6, 2023

medoedoff commented Nov 3, 2023

lwr20 commented Nov 3, 2023 • edited Loading

barsilver commented Aug 3, 2022 •

edited

Loading

barsilver commented Aug 6, 2022 •

edited

Loading

lwr20 commented Aug 23, 2022 •

edited

Loading

lwr20 commented Nov 3, 2023 •

edited

Loading