Vault Agent Injector not working on EKS 1.25

[dancook993]

Hello,

I have a three node vault cluster with raft storing running hashicorp/vault:1.8.0 on my EKS production cluster. In my production cluster, I have a vault agent injector running vault-k8s:0.11.0 which is succesfully mounting secrets into pods. The EKS version of this cluster is 1.22.

In my staging cluster, I then have a vault agent injector running vault-k8s:0.11.0 (Vault helm 0.11.0). This connects to the production vault via its public ingress name. The EKS version of this cluster is 1.25. We have upgraded from 1.21 -> 1.25 and somewhere during this upgrade has broken vault agent injecting secrets into pods.

The logs I see in the stage vault agent injector are:
2024-07-15T18:20:15.104Z [INFO] handler: Starting handler..
Listening on ":8080"...
2024-07-15T18:20:15.188Z [INFO] handler.auto-tls: Generated CA
2024-07-15T18:20:15.188Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2024-07-15T18:20:36.532Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:20:40.768Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:21:06.087Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:21:06.926Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:21:10.379Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:21:35.591Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:22:07.043Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:22:39.532Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:23:05.544Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:23:07.980Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:24:43.173Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2024-07-15T18:25:00.118Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s

I have tried looking in the EKS api-server logs to look for any errors with the mutate requests but these seem to be passing as expected. Nothing has changed from either of our vault deployments other than updating the EKS version.

The mutating web hook configuration looks like this:
webhooks:

• admissionReviewVersions:
  • v1beta1
  • v1
    clientConfig:
    caBundle: {REDACTED}
    service:
    name: vault-agent-injector-svc
    namespace: app
    path: /mutate
    port: 443
    failurePolicy: Ignore
    matchPolicy: Exact
    name: vault.hashicorp.com
    namespaceSelector: {}
    objectSelector: {}
    reinvocationPolicy: Never
    rules:
  • apiGroups:
    • ""
      apiVersions:
    • v1
      operations:
    • CREATE
    • UPDATE
      resources:
    • pods
      scope: '*'
      sideEffects: None
      timeoutSeconds: 30

The pod where we are trying to have the secret mounted has the following annotations:
vault.hashicorp.com/agent-configmap: secrets-updater
vault.hashicorp.com/agent-inject: true

These are the same annotations used in the production vault agent injector where it is working

Does anyone where where is best to log for further errors or information? I thought the kube-api server may be the best place but didn't see any mutate errors there. Without the vault agent giving any errors it is very difficult to troubleshoot. Setting the log level to debug also doesn't help.

[dancook993]

Maybe this could be due to the service account changes from EKS 1.23 -> 1.24 as the agent injector seems to be able to mutate correctly.