Reconcile Authorino when OSSM

[guicassolato]

• Reconciles Authorino when the service mesh runtime is OpenShift Service Mesh (OSSM)
  • Tries to find the IstioOperator first
  • Falls back to ServiceMeshControlPlane when there's no match for kind IstioOperator
  • Adds the extension provider to IstioOperator or ServiceMeshControlPlane
  • Registers the kuadrant-system to the mesh (OSSM only)
• Reverts the following PRs
  • Only reconcile Authorino if Istio #104
  • consider NoKindMatchError when reading IstioOperator #106
  • disable status check for authorino when not istio #110

Related to #68.

---

Verification steps

Prerequisites:

• OpenShift cluster
• oc CLI tool
• Admin privileges to the OpenShift cluster
• The following operators installed in the cluster
  • OpenShift Elasticsearch
  • Red Hat OpenShift distributed tracing platform
  • Kiali
  • Red Hat OpenShift Service Mesh

Login:

export CLUSTER_DOMAIN=<openshift-cluster-domain>
oc login --token=<secret> --server=https://api.$CLUSTER_DOMAIN:6443

Install Gateway API:

kubectl get crd gateways.gateway.networking.k8s.io || { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.4.0" | kubectl apply -f -; }

Deploy the service mesh control plane:

kubectl create namespace istio-system
kubectl apply -n istio-system -f -<<EOF
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
  name: istiocontrolplane # set the ISTIOOPERATOR_NAME env var in the Kuadrant Operator deployment if this is named otherwise
spec:
  runtime:
    components:
      pilot:
        container:
          env:
            PILOT_ENABLE_GATEWAY_API: "true"
            PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER: "true"
            PILOT_ENABLE_GATEWAY_API_STATUS: "true"
  version: v2.3
  policy:
    type: Istiod
  tracing:
    type: Jaeger
    sampling: 10000
  telemetry:
    type: Istiod
  addons:
    prometheus:
      enabled: true
    jaeger:
      name: jaeger
      install:
        storage:
          type: Memory
    kiali:
      enabled: true
      name: kiali
    grafana:
      enabled: true
EOF

Deploy the Istio gateway:

kubectl apply -n istio-system -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  labels:
    istio: ingressgateway
  name: istio-ingressgateway
spec:
  gatewayClassName: istio
  listeners:
  - name: default
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  addresses:
  - value: istio-ingressgateway.istio-system.svc.cluster.local
    type: Hostname
EOF

Deploy the API to be protected (toystore):

kubectl create namespace toystore
kubectl apply -n toystore -f https://github.com/raw/Kuadrant/kuadrant-operator/3e05cf590dab0faeca63c4b13b88771676beddc2/examples/toystore/toystore.yaml

Add the API namespace to the mesh:

kubectl apply -n istio-system -f -<<EOF
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
  name: default
  namespace: istio-system
spec:
  members:
  - toystore
EOF
kubectl patch -n toystore deployment/toystore -p '{"spec": {"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}} }'

Connect the API to the gateway:

kubectl apply -n toystore -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: toystore
  labels:
    app: toystore
spec:
  parentRefs:
    - name: istio-ingressgateway
      namespace: istio-system
  hostnames: ["*.toystore.apps.$CLUSTER_DOMAIN"]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/toy"
          method: GET
      backendRefs:
        - name: toystore
          port: 80
EOF

Install the Kuadrant Operator:

make install
make deploy IMG=quay.io/guicassolato/kuadrant:kuadrant-operator-ossm-20221129

Deploy the Kuadrant instance:

kubectl apply -n kuadrant-system -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

Expose the API for N/S traffic:

oc expose -n istio-system service/istio-ingressgateway --port 8080 --name toystore --hostname=api.toystore.apps.$CLUSTER_DOMAIN

Try the API unprotected:

curl http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i #=> 200 OK

Create an AuthPolicy:

kubectl apply -n toystore -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: AuthPolicy
metadata:
  name: toystore
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  authScheme:
    identity:
    - name: apikey
      apiKey:
        selector:
          matchLabels:
            app: toystore
        allNamespaces: true # <=== this is needed because kuadrant creates the authconfig in the kuadrant-system namespace instead of in the same namespace as the authpolicy
      credentials:
        in: authorization_header
        keySelector: APIKEY
EOF

Try the API while missing the API key:

curl http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i #=> 401 Unauthorized

Create an API key:

kubectl apply -n toystore -f -<<EOF
apiVersion: v1
kind: Secret
metadata:
  name: api-key-1
  labels:
    authorino.kuadrant.io/managed-by: authorino
    app: toystore
stringData:
  api_key: ndyBzreUzF4zqDQsqSPMHkRhriEOtcRx
type: Opaque
EOF

Try the API with a valid the API key:

curl -H 'Authorization: APIKEY ndyBzreUzF4zqDQsqSPMHkRhriEOtcRx' http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i #=> 200 OK

[pehala]

• Registers the kuadrant-system to the mesh (OSSM only)

Does this mean that Kuadrant will add ServiceMeshMember to kuadrant-system namespace?

[guicassolato]

• Registers the kuadrant-system to the mesh (OSSM only)

Does this mean that Kuadrant will add ServiceMeshMember to kuadrant-system namespace?

It does (indirectly). Actually we add the kuadrant-system namespace to the default ServiceMeshMemberRoll resource – creating it if it doesn't exist yet. It's then the service mesh member roll controller that creates the ServiceMeshMember resource.

[pehala]

It does (indirectly). Actually we add the kuadrant-system namespace to the default ServiceMeshMemberRoll resource – creating it if it doesn't exist yet. It's then the service mesh member roll controller that creates the ServiceMeshMember resource.

How do you know to which service mesh add the namespace? OSSM supports multiple meshes on the same cluster

[guicassolato]

It does (indirectly). Actually we add the kuadrant-system namespace to the default ServiceMeshMemberRoll resource – creating it if it doesn't exist yet. It's then the service mesh member roll controller that creates the ServiceMeshMember resource.

How do you know to which service mesh add the namespace? OSSM supports multiple meshes on the same cluster

Right now, Kuadrant is only aware of one service mesh. The implementation finds the ServiceMeshControlPlane resource by relying on the same existing environment variables currently used to point the Istio control plane, i.e. ISTIOOPERATOR_NAMESPACE and ISTIOOPERATOR_NAME.

Yesterday it was proposed in the Kuadrant tech discussion meeting to make this explicit in the Kuadrant API. I suppose that would be a good opportunity to expand to multiple service meshes in the same cluster?

Implementation-wise, it's not a big deal. We can iterate over the list of service mesh control planes in the Kuadrant CR and inject the Kuadrant instance in each one of them. It could look something like the following maybe:

apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec:
  meshes:
  - group: install.istio.io/v1alpha1
    kind: IstioOperator
    name: istiocontrolplane
    namespace: istio-system
  - group: maistra.io/v2
    kind: ServiceMeshControlPlane
    name: basic
    namespace: ossm-system
  - group: maistra.io/v2
    kind: ServiceMeshControlPlane
    name: enterprise
    namespace: ossm-system

Anyway, this for a future iteration. Not for this PR.

[pehala]

Thanks for explanation :)

[alexsnaps]

I'm in favor of merging this… to main actually. I understand that's a heuristic we'd say we'd rather not have. But we can open the RFC to change the Kuadrant CR accordingly. But in the meantime this would work in both environments…

That's a great milestone for v0.1 I think…

[eguzki]

Looks good.

I am not entirely happy with the rules implemented to find out which mesh is installed in the cluster. What if OSSM is up and running but IstioOperator CRD is installed?

Having to add maistra api's code to our repo is a bad smell. Any good reason to do so?

Documentation?

Plans for testing?

[eguzki]

why do we need this code in the kuadrant operator repo?
why not import them?

[guicassolato]

I'm afraid the types cannot be imported directly.

The packages have deep dependencies that are based on old golang version and libraries not compatible with the version of our code. Luckily we don't depend on those incompatible bit as we only need the type definitions.

For the record, this wasn't my first option either, but it turned out to be the best approach until Maistra team upgrades to newer version of Operator SDK, something I was told to be in their roadmap.

BTW, the approach of hosting copies the types is adopted by https://github.com/maistra/istio-operator itself.

[eguzki]

nitpick: iopName stands for IstioOperator (object) Name. Maybe we should find another name.