Envoygateway wasm controller

[eguzki]

What

Controller to configure kuadrant wasm-shim when EnvoyGateway is the gateway provider.

It enables kuadrant rate limiting

Verification Steps

① Setup (Persona: Cluster admin)

make local-setup GATEWAYAPI_PROVIDER=envoygateway

Request an instance of Kuadrant in the kuadrant-system namespace:

kubectl -n kuadrant-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

② Deploy the Toy Store sample application (Persona: App developer)

kubectl apply -f examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: eg
    namespace: envoy-gateway-system
  hostnames:
  - api.example.com
  rules:
  - matches:
    - method: GET
      path:
        type: PathPrefix
        value: "/toys"
    backendRefs:
    - name: toystore
      port: 80
EOF

Wait for the deployment:

kubectl wait --timeout=5m deployment/toystore --for=condition=Available

Export the gateway hostname and port:

export INGRESS_HOST=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.status.addresses[0].value}')
export INGRESS_PORT=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.spec.listeners[?(@.name=="http")].port}')

Test the route with traffic generator of 1 request per second.

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null --resolve api.example.com:$INGRESS_PORT:$INGRESS_HOST "http://api.example.com:$INGRESS_PORT/toys" -i | grep -E --color "\b(429)\b|$"; sleep 1; done

All responses should be 200 OK.

③ Protect the Toy Store application (Persona: App developer)

In other terminal, create the RateLimitPolicy for all the traffic defined for the HTTPRoute, max 5 request every 10 seconds.

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: toystore
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    basic:
      rates:
      - limit: 5
        duration: 10
        unit: second
EOF

Check the response of the traffic being run in the previous terminal. It should now be rate limited to 5 request every 10 seconds.

④ Verification of kuadrant managed EnvoyGateway resources

EnvoyExtensionPolicy

Check EnvoyExtensionPolicy resource defined by kuadrant

kubectl get envoyextensionpolicy -n envoy-gateway-system kuadrant-wasm-for-eg -o yaml | yq e -P

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  annotations:
    kuadrant.io/namespace: kuadrant-system
  creationTimestamp: "2024-09-09T14:32:01Z"
  generation: 1
  name: kuadrant-wasm-for-eg
  namespace: envoy-gateway-system
  ownerReferences:
    - apiVersion: gateway.networking.k8s.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: Gateway
      name: eg
      uid: e361aaf2-83e8-4cb6-b9e7-eba5b8870744
  resourceVersion: "3274"
  uid: e1f2a276-bf6f-4d7b-a759-8ddccd1f872d
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  wasm:
    - code:
        image:
          url: oci://quay.io/kuadrant/wasm-shim:latest
        type: Image
      config:
        failureMode: deny
        rateLimitPolicies:
          - domain: default/toystore
            hostnames:
              - api.example.com
            name: default/toystore
            rules:
              - conditions:
                  - allOf:
                      - operator: startswith
                        selector: request.url_path
                        value: /toys
                      - operator: eq
                        selector: request.method
                        value: GET
                data:
                  - static:
                      key: limit.basic__48de662e
                      value: "1"
            service: kuadrant-rate-limiting-service
      failOpen: false
      name: kuadrant-wasm-shim
      rootID: kuadrant_wasm_shim
status:
  ancestors:
    - ancestorRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eg
        namespace: envoy-gateway-system
      conditions:
        - lastTransitionTime: "2024-09-09T14:32:03Z"
          message: Policy has been accepted.
          reason: Accepted
          status: "True"
          type: Accepted
      controllerName: gateway.envoyproxy.io/gatewayclass-controller

Few things to highlight:

• The envoyextensionpolicy targets the gateway.
• Wasm filter defined with a reference to the kuadrant's wasm-shim OCI image oci://quay.io/kuadrant/wasm-shim:latest.
• The wasm configuration is exactly the same as the one defined for Istio's wasmplugin resource.
• EnvoyGateway fills the status, so kuadrant controller can test status for the rate limit policy enforce's condition (missing, to be done in other PR).

EnvoyPatchPolicy

Check EnvoyPatchPolicy resource defined by kuadrant

The EnvoyPatchPolicy is used to tell Envoy the address of limitador, which is being used by the wasm module to open a grpc connection. Basically, it needs to tell: Envoy cluster kuadrant-rate-limiting-service is the address limitador-limitador.kuadrant-system.svc.cluster.local:8081

kubectl get envoypatchpolicy -n envoy-gateway-system kuadrant-patch-for-kuadrant-wasm-for-eg -o yaml | yq e -P

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  creationTimestamp: "2024-09-09T14:32:01Z"
  generation: 1
  name: kuadrant-patch-for-kuadrant-wasm-for-eg
  namespace: envoy-gateway-system
  ownerReferences:
    - apiVersion: gateway.envoyproxy.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: EnvoyExtensionPolicy
      name: kuadrant-wasm-for-eg
      uid: e1f2a276-bf6f-4d7b-a759-8ddccd1f872d
  resourceVersion: "3277"
  uid: cf7d60fb-fb44-4b5e-b891-5257fda80038
spec:
  jsonPatches:
    - name: kuadrant-rate-limiting-service
      operation:
        op: add
        path: ""
        value:
          connect_timeout: 1s
          http2_protocol_options: {}
          lb_policy: ROUND_ROBIN
          load_assignment:
            cluster_name: kuadrant-rate-limiting-service
            endpoints:
              - lb_endpoints:
                  - endpoint:
                      address:
                        socket_address:
                          address: limitador-limitador.kuadrant-system.svc.cluster.local
                          port_value: 8081
          name: kuadrant-rate-limiting-service
          type: STRICT_DNS
      type: type.googleapis.com/envoy.config.cluster.v3.Cluster
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
status:
  ancestors:
    - ancestorRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eg
        namespace: envoy-gateway-system
      conditions:
        - lastTransitionTime: "2024-09-09T14:32:03Z"
          message: Policy has been accepted.
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2024-09-09T14:32:03Z"
          message: Patches have been successfully applied.
          reason: Programmed
          status: "True"
          type: Programmed
      controllerName: gateway.envoyproxy.io/gatewayclass-controller

[codecov]

Codecov Report

Attention: Patch coverage is 77.47748% with 100 lines in your changes missing coverage. Please review.

Please upload report for BASE (envoygateway@9d29fcc). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...llers/envoygateway_limitador_cluster_controller.go	78.04%	17 Missing and 10 partials ⚠️
controllers/envoygateway_wasm_controller.go	82.30%	13 Missing and 10 partials ⚠️
pkg/envoygateway/mutators.go	36.11%	14 Missing and 9 partials ⚠️
pkg/rlptools/wasm/utils.go	88.05%	4 Missing and 4 partials ⚠️
pkg/library/mappers/kuadrant_list_mapper.go	50.00%	4 Missing and 2 partials ⚠️
pkg/rlptools/wasm/types.go	50.00%	3 Missing and 3 partials ⚠️
pkg/envoygateway/utils.go	80.00%	2 Missing and 2 partials ⚠️
pkg/library/gatewayapi/utils.go	40.00%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##             envoygateway     #848   +/-   ##
===============================================
  Coverage                ?   81.73%           
===============================================
  Files                   ?       87           
  Lines                   ?     6927           
  Branches                ?        0           
===============================================
  Hits                    ?     5662           
  Misses                  ?      848           
  Partials                ?      417           

Flag	Coverage Δ
bare-k8s-integration	5.20% <11.03%> (?)
controllers-integration	73.72% <76.57%> (?)
envoygateway-integration	47.47% <71.62%> (?)
gatewayapi-integration	11.01% <11.03%> (?)
istio-integration	51.64% <25.67%> (?)
unit	28.24% <0.46%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	71.42% <0.00%> (?)
api/v1beta2 (u)	85.35% <0.00%> (?)
pkg/common (u)	88.13% <0.00%> (?)
pkg/istio (u)	70.95% <0.00%> (?)
pkg/log (u)	94.73% <0.00%> (?)
pkg/reconcilers (u)	∅ <0.00%> (?)
pkg/rlptools (u)	84.26% <0.00%> (?)
controllers (i)	83.24% <0.00%> (?)

Files with missing lines	Coverage Δ
...llers/rate_limiting_istio_wasmplugin_controller.go	73.78% <100.00%> (ø)
controllers/test_common.go	100.00% <100.00%> (ø)
pkg/istio/utils.go	79.16% <100.00%> (ø)
pkg/rlptools/utils.go	100.00% <100.00%> (ø)
pkg/library/gatewayapi/utils.go	87.15% <40.00%> (ø)
pkg/envoygateway/utils.go	71.42% <80.00%> (ø)
pkg/library/mappers/kuadrant_list_mapper.go	50.00% <50.00%> (ø)
pkg/rlptools/wasm/types.go	46.66% <50.00%> (ø)
pkg/rlptools/wasm/utils.go	87.55% <88.05%> (ø)
controllers/envoygateway_wasm_controller.go	82.30% <82.30%> (ø)
... and 2 more

[guicassolato]

I was banging my head against the wall for 20 min or so, trying to find where in the code we "fixed" the imaginary route thing for RLPs. It turns out we haven't. 🙄

Leaving this comment here then only so we remember to fix the issue here as well whenever it is addressed with #688.

IOW, this LOC is wrong, but consistently wrong with all the other places where we made the same mistake, making it kind of right, if it makes any sense 😵‍💫

[eguzki]

good point. Thanks for that. That will ease the process of rebasing when we fix that in main. With a little bit of luck, we can merge this into main before we fix it and then no rebase conflicts.

[eguzki]

@adam-cattermole ready for review

[adam-cattermole]

Nice job here, lgtm

[adam-cattermole]

This is interesting, I guess if we document support for envoygateway in the repo / on the website we should call out this requirement to enable the envoy patch policy API

[eguzki]

Sure!