AuthPolicy Atomic Overrides

[Boomatang]

AuthPolicy Atomic Overrides

Closes: #464

Verification

Base Setup

Create local cluster from this branch

make local-setup

Apply Kuadrant CR.

echo "  
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system
spec: {}
"

Install the toy store app

kubectl apply -f examples/toystore/toystore.yaml

Add HTTP Route

echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: istio-ingressgateway
    namespace: istio-system
  hostnames:
  - api.toystore.com
  rules:
  - matches:
    - path:
        type: Exact
        value: "/toy"
      method: GET
    backendRefs:
    - name: toystore
      port: 80
" | kubectl apply -f -

Create secrets for Auth. The secrets are doubled to be created in two namespaces.

echo "
apiVersion: v1
kind: Secret
metadata:
  name: alice-key
  namespace: default
  labels:
    authorino.kuadrant.io/managed-by: authorino
    app: toystore
  annotations:
    secret.kuadrant.io/user-id: alice
stringData:
  api_key: IAMALICE
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: alice-key
  namespace: istio-system
  labels:
    authorino.kuadrant.io/managed-by: authorino
    app: toystore
  annotations:
    secret.kuadrant.io/user-id: alice
stringData:
  api_key: IAMALICE
type: Opaque
" | kubectl apply -f -

Set up gateway url

export INGRESS_HOST=$(kubectl get gtw istio-ingressgateway -n istio-system -o jsonpath='{.status.addresses[0].value}')
export INGRESS_PORT=$(kubectl get gtw istio-ingressgateway -n istio-system -o jsonpath='{.spec.listeners[?(@.name=="http")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Set up a watch on AuthConfigs.

watch kubectl get authconfig -A

Case 1

There is an existing AuthPolicy attached to a HTTPRoute and the system admin attaches a AuthPolicy to the Gateway with atonic overrides.

Create HTTPRoute AuthPolicy. Expect watch to contain one AuthConfig ap-default-toystore

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: TOYSTORE
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f - 

Check auth is working. Note the name of the Authorization key, TOYSTORE. Expected result is 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Add the Gateway AuthPolicy that contains overrides. Expect AuthConfig to change to ap-istio-system-gateway

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  overrides:
    rules:
      authentication:
        "api-key-users":
          apiKey:
            selector:
              matchLabels:
                app: toystore
            allNamespaces: true
          credentials:
            authorizationHeader:
              prefix: APIKEY
      response:
        success:
          dynamicMetadata:
            "identity":
              json:
                properties:
                  "userid":
                    selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f -

Curl the endpoint again with Route Key. Expected 401

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

This show the Gateway overrides are working.

Case 2

In this case the Gateway AuthPolicy has being defined before any route AuthPolicies have being attached.

Set the cluster back to the configuration as per the Base Setup.

Install the gateway AuthPolicy. Expected AuthConfig on watch: ap-istio-system-gateway

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  overrides:
    rules:
      authentication:
        "api-key-users":
          apiKey:
            selector:
              matchLabels:
                app: toystore
            allNamespaces: true
          credentials:
            authorizationHeader:
              prefix: APIKEY
      response:
        success:
          dynamicMetadata:
            "identity":
              json:
                properties:
                  "userid":
                    selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f -

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Apply Auth Policy to route. Expected AuthConfig on watch: ap-istio-system-gateway

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: TOYSTORE
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f - 

Curl the endpoint with Route Key. Expected 401

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

This show the Gateway overrides are working for routes added after the initial overrides.

Case 3

The cluster has exiting AuthPolicies attached to the routes and gateways. The system admin later defines the overrides on the gateway AuthPolicy.

Set the cluster back to the configuration as per the Base Setup.

Create the Route and Gateway AuthPolices. Note that the Gateway AuthPolicy does not define any overrides. Expected AuthConfig in watch: ap-default-toystore

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: TOYSTORE
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
---
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: APIKEY
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f - 

Curl the endpoint with Route Key. Expected 200

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 401.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Update the Gateway AuthPolicy to have overrides. Expected AuthConfig on watch: ap-istio-system-gateway.

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  overrides:
    rules:
      authentication:
        "api-key-users":
          apiKey:
            selector:
              matchLabels:
                app: toystore
            allNamespaces: true
          credentials:
            authorizationHeader:
              prefix: APIKEY
      response:
        success:
          dynamicMetadata:
            "identity":
              json:
                properties:
                  "userid":
                    selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f -

Curl the endpoint with Route Key. Expected 401

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

This proves that existing Gateway AuthPolicies can be updated to contain overrides.

Case 4

The cluster in this case is configure to have overrides on the Gateway from the start. At a later time the system admin removes the overrides from the gateway AuthPolicy.

Set the cluster back to the configuration as per the Base Setup.

Create the Route and Gateway AuthPolices. Note that the Gateway AuthPolicy does define overrides. Expected AuthConfig in watch: ap-istio-system-gateway

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: TOYSTORE
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
---
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  overrides:
      rules:
        authentication:
          "api-key-users":
            apiKey:
              selector:
                matchLabels:
                  app: toystore
              allNamespaces: true
            credentials:
              authorizationHeader:
                prefix: APIKEY
        response:
          success:
            dynamicMetadata:
              "identity":
                json:
                  properties:
                    "userid":
                      selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f - 

Curl the endpoint with Route Key. Expected 401

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 200.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Update the Gateway AuthPolicy to remove the overrides. Expect AuthConfig in watch: ap-default-toystore

echo "
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: gateway
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  rules:
    authentication:
      "api-key-users":
        apiKey:
          selector:
            matchLabels:
              app: toystore
          allNamespaces: true
        credentials:
          authorizationHeader:
            prefix: APIKEY
    response:
      success:
        dynamicMetadata:
          "identity":
            json:
              properties:
                "userid":
                  selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
" | kubectl apply -f - 

Curl the endpoint with Route Key. Expected 200

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: TOYSTORE IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

Curl endpoint with key defined in the Gateway AuthPolicy, APIKEY. Expected 401.

curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -H 'Host: api.toystore.com' http://$GATEWAY_URL/toy

This proves AuthPolicies can be updated to remove overrides.````

[guicassolato]

Please see https://github.com/Kuadrant/kuadrant-operator/pull/523/files#r1559122844

[guicassolato]

Nice. I wonder if we could do something similar (or exactly the same!) for the RLP as well.

The RLP doesn't have the same 1:1 relation to the underlying resources as the AP does with the AuthConfigs, so it would somewhere else of course. But useful for setting the status of the policy later on nevertheless.

[KevFan]

Yes, I think we can and probably should do something similar for RLPs also 👍

[KevFan]

In the RLP equivalent, I've opted to add a CEL validation to only allow overrides block for policies targeting a gateway. Not sure do we want to do the same here

Suggested change

	// +kubebuilder:validation:XValidation:rule="!(has(self.overrides) && has(self.defaults))",message="Explicit overrides and explicit defaults are mutually exclusive"
	// +kubebuilder:validation:XValidation:rule="!(has(self.overrides) && has(self.defaults))",message="Explicit overrides and explicit defaults are mutually exclusive"
	// +kubebuilder:validation:XValidation:rule="!(has(self.overrides) && self.targetRef.kind != 'Gateway')",message="Overrides are only allowed for policies targeting a Gateway resource"

[Boomatang]

I was thinking about this, and I felt if we blocked the overrides on the route policies, we should block the defaults on the gateway policies. I do think we should be consistent on what we allow on both sets of policies and that it is easier to add access than remove it at a later date. What do you think @KevFan

[guicassolato]

we should block the defaults on the gateway policies.

Sorry, @Boomatang. Maybe I'm not following. The defaults on the gateway is a valid use case. I don't think we want to block them. How would one set defaults for multiple routes?

[Boomatang]

I am working off my understand of the hierarchy chart, the defaults sets value for the resources above in the tree and overrides set values on resources lower in the tree. As we don't support setting these values on resources about the gateway, we don't need to allow defaults on it.

I did check the RFC0009 and it would allow setting the defaults as we currently have it but that seems to be different than the definition in the gateway-api GEP2649. I know that GEP is being up dated, so maybe that definition is also being updated.

[guicassolato]

Not really. Both defaults and overrides are set for the resources lower in the tree. The arrows (up or down) tell which level have precedence in case of conflicting rules set at multiple levels. (The arrow pointing from higher precedence to lower.)

[Boomatang]

During the sync we cleared this up. I will be implementing the CEL to disable overrides on the policy attached to the HttpRoutes.

[KevFan]

Verified, changes looks good to me 👍

[guicassolato]

Maybe add a comment here that this only works because of single policy of a kind allowed per target object. Since the gateway policy was skipped in line 186, all the other refs are of policies targeting routes. In the future, that may no longer be the case.

[Boomatang]

I see your point, but I don't think a comment is worth it. I would assume there are other spots in the code base that assume only one type of resources is attached to the gateway. When the day comes when we support it, I would expect that would be a large change to the whole code base and not just here.

[guicassolato]

Maybe rename ap here as otherAP, to avoid confusion with the AP that triggered the reconciliation?

[guicassolato]

This looks very good, @Boomatang. Thank you!

Leaving here almost as a self reminder. In the future, we can revisit this and try to leverage more the gatewayapi.Topology and gatewayapi.TopologyIndexes (aka DAG) as an alternative to nested API reads such as:

kuadrant-operator/controllers/authpolicy_authconfig.go

Lines 192 to 204 in 7efe3c9

	gw := &gatewayapiv1.Gateway{}
	namespace := ptr.Deref(parentRef.Namespace, gatewayapiv1.Namespace(route.GetNamespace()))
	err := c.Get(ctx, client.ObjectKey{Name: string(parentRef.Name), Namespace: string(namespace)}, gw)
	if err != nil {
	return false, err
	}
	annotation, ok := gw.GetAnnotations()[common.AuthPolicyBackRefAnnotation]
	if !ok {
	continue
	}
	otherAP := &api.AuthPolicy{}
	err = c.Get(ctx, utils.NamespacedNameToObjectKey(annotation, gw.Namespace), otherAP)

[guicassolato]

Some additional verification steps I tried, for the CEL validation piece:

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  rules:
    authentication: {}
  overrides:
    rules:
      authentication: {}
EOF
The AuthPolicy "my-policy" is invalid:
* spec: Invalid value: "object": Implicit defaults and explicit overrides are mutually exclusive
* spec: Invalid value: "object": Overrides are not allowed for policies targeting a HTTPRoute resource

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  defaults:
    rules:
      authentication: {}
  overrides:
    rules:
      authentication: {}
EOF
The AuthPolicy "my-policy" is invalid:
* spec: Invalid value: "object": Explicit overrides and explicit defaults are mutually exclusive
* spec: Invalid value: "object": Overrides are not allowed for policies targeting a HTTPRoute resource

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
    namespace: istio-system
  rules:
    authentication: {}
  overrides:
    rules:
      authentication: {}
EOF
The AuthPolicy "my-policy" is invalid: spec: Invalid value: "object": Implicit defaults and explicit overrides are mutually exclusive

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
    namespace: istio-system
  defaults:
    rules:
      authentication: {}
  overrides:
    rules:
      authentication: {}
EOF
The AuthPolicy "my-policy" is invalid: spec: Invalid value: "object": Explicit overrides and explicit defaults are mutually exclusive

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
    namespace: istio-system
  overrides:
    rules:
      authentication: {}
    routeSelectors:
    - matches:
      - method: GET
EOF
The AuthPolicy "my-policy" is invalid: spec: Invalid value: "object": route selectors not supported when targeting a Gateway

$ kubectl apply -f -<<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: my-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
    namespace: istio-system
  overrides:
    rules:
      authentication:
        anonymous-access:
          anonymous: {}
          routeSelectors:
          - matches:
            - method: GET
EOF
The AuthPolicy "my-policy" is invalid: spec: Invalid value: "object": route selectors not supported when targeting a Gateway