Fix override RLPs on multiple gateway parents

[guicassolato]

Before:

We are currently wrongly generating the wasm configs for cases of multiple gateway parents linked to routes, having override RateLimitPolicies targeting the gateways and an overridden route policy.

The overridden policy config ends up in the WasmPlugin and in the Limitador CR with mismatching domain/namespace, causing the override policy to never be enforced – in spite of the status reporting otherwise.

After:

This PR simplifies the configuration of overridden RateLimitPolicies.

A route RateLimitPolicy overridden at a particular gateway will no longer be added to the gateway's WasmPlugin config; rather, the override gateway policy will, with its domain value kept unchanged, as if all routes under the gateway were not targeted by policies.

In the Limitador CR, all RateLimitPolicies (override and overridden) will be added, even when some never get triggered (due to been overridden.)

Verification

Setup

make local-setup

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gw
  namespace: kuadrant-system
spec:
  gatewayClassName: istio
  addresses:
  - type: IPAddress
    value: "172.18.0.10"
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    name: api
    hostname: "*.external"
    port: 80
    protocol: HTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-gw
  namespace: kuadrant-system
spec:
  gatewayClassName: istio
  addresses:
  - type: IPAddress
    value: "172.18.0.11"
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    name: api
    hostname: "*.internal"
    port: 80
    protocol: HTTP
EOF

kubectl apply -f https://github.com/raw/Kuadrant/kuadrant-operator/main/examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: toystore-api
spec:
  parentRefs:
  - name: external-gw
    namespace: kuadrant-system
  - name: internal-gw
    namespace: kuadrant-system
  hostnames:
  - api.external
  - api.internal
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: "/cars"
    - path:
        type: PathPrefix
        value: "/dolls"
    backendRefs:
    - name: toystore
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: "/admin"
    backendRefs:
    - name: toystore
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: toystore-www
spec:
  parentRefs:
  - name: external-gw
    namespace: kuadrant-system
  - name: internal-gw
    namespace: kuadrant-system
  hostnames:
  - www.external
  - www.internal
  rules:
  - backendRefs:
    - name: toystore
      port: 80
EOF

Send requests

Repeat the following requests as many times as needed after each state transition between creating and updating policies:

curl --resolve api.external:80:172.18.0.10 "http://api.external/cars"

curl --resolve api.internal:80:172.18.0.11 "http://api.internal/cars"

curl --resolve www.external:80:172.18.0.10 "http://www.external/cars"

curl --resolve www.internal:80:172.18.0.11 "http://www.internal/cars"

What to look at in the responses:

• Without any policies → all requests should go through without any header modification or limits
• After creating AuthPolicies → a special request header authpolicy is injected containing the name of the policy that was enforced
• After creating RateLimitPolicies → both gateway policies (external: 3rp10s, internal: 5rp10s) must be enforced at both routes respectively for *.external and *.internal traffic
• After modifying the external override gateway RateLimitPolicy to defaults → internal override gateway policy (5rp10s) must continue to be enforced at both routes for *.internal traffic; toystore api specific policy (1rp10s) must be enforced at api.external traffic, while external gateway override (3rp10s) continues to be enforced at the rest of *.external (i.e. www.external)

Create policies

kubectl -n kuadrant-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: external-gw-auth
  namespace: kuadrant-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external-gw
  overrides:
    rules:
      response:
        success:
          headers:
            "authpolicy":
              plain:
                value: external-gw-auth
---
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: internal-gw-auth
  namespace: kuadrant-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: internal-gw
  overrides:
    rules:
      response:
        success:
          headers:
            "authpolicy":
              plain:
                value: internal-gw-auth
---
apiVersion: kuadrant.io/v1beta2
kind: AuthPolicy
metadata:
  name: toystore-api-auth
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore-api
  rules:
    response:
      success:
        headers:
          "authpolicy":
            plain:
              value: toystore-api-auth
EOF

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: external-gw-rl
  namespace: kuadrant-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external-gw
  overrides:
    limits:
      "external-gw-3rp10s":
        rates:
        - limit: 3
          duration: 10
          unit: second
---
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: internal-gw-rl
  namespace: kuadrant-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: internal-gw
  overrides:
    limits:
      "internal-gw-5rp10s":
        rates:
        - limit: 5
          duration: 10
          unit: second
---
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: toystore-api-rl
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore-api
  limits:
    "toystore-api-1rp10s":
      rates:
      - limit: 1
        duration: 10
        unit: second
EOF

Change a gateway policy from overrides → defaults

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: external-gw-rl
  namespace: kuadrant-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: external-gw
  defaults:
    limits:
      "external-gw-3rp10s":
        rates:
        - limit: 3
          duration: 10
          unit: second
EOF

[codecov]

Codecov Report

Attention: Patch coverage is 81.77083% with 35 lines in your changes are missing coverage. Please review.

Project coverage is 82.81%. Comparing base (ece13e8) to head (8c489c5).
Report is 108 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
+ Coverage   80.20%   82.81%   +2.60%     
==========================================
  Files          64       73       +9     
  Lines        4492     5737    +1245     
==========================================
+ Hits         3603     4751    +1148     
- Misses        600      652      +52     
- Partials      289      334      +45     

Flag	Coverage Δ
bare-k8s-integration	4.42% <0.00%> (?)
controllers-integration	72.29% <81.77%> (?)
gatewayapi-integration	10.98% <0.00%> (?)
integration	?
istio-integration	56.11% <81.77%> (?)
unit	32.56% <4.16%> (+2.52%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	71.42% <ø> (ø)
api/v1beta2 (u)	93.58% <100.00%> (+2.16%)	⬆️
pkg/common (u)	88.13% <ø> (-0.70%)	⬇️
pkg/istio (u)	75.09% <ø> (+1.17%)	⬆️
pkg/log (u)	94.73% <ø> (ø)
pkg/reconcilers (u)	∅ <ø> (∅)
pkg/rlptools (u)	81.59% <ø> (+2.13%)	⬆️
controllers (i)	81.58% <78.98%> (+4.78%)	⬆️

Files	Coverage Δ
controllers/ratelimitpolicy_controller.go	82.69% <100.00%> (+8.69%)	⬆️
controllers/target_status_controller.go	89.34% <100.00%> (ø)
...library/kuadrant/apimachinery_status_conditions.go	96.36% <100.00%> (+0.53%)	⬆️
pkg/library/reconcilers/target_ref_reconciler.go	79.41% <ø> (+1.96%)	⬆️
pkg/rlptools/rate_limit_index.go	76.27% <ø> (+5.08%)	⬆️
pkg/rlptools/utils.go	100.00% <100.00%> (ø)
pkg/rlptools/wasm_utils.go	79.35% <100.00%> (+0.54%)	⬆️
controllers/rate_limiting_wasmplugin_controller.go	76.69% <75.00%> (ø)
pkg/rlptools/overrides.go	88.00% <88.00%> (ø)
controllers/ratelimitpolicy_limits.go	81.03% <81.13%> (-1.23%)	⬇️
... and 1 more

... and 25 files with indirect coverage changes

[guicassolato]

I found that debugging reconciliation calls can be a PITA, so I'm slowly adding these to the functions.

[KevFan]

Verified, working as expected 🏅 Changes looks good to me 👍