PV not being created upon provisioning - resulting in Vault Pod being stuck in pending state

[BJWRD]

Describe the bug
Upon Helm deployment using the values referenced below, there doesn't appear to be a PV being deployed alongside the PVC (not bound). This is resulting in the Vault pod being stuck in a pending state.

To Reproduce
Any deployment including the following Helm values -

values:
    server:
      ingress:
        enabled: true
        hosts:
          - host: hostname.com
      dataStorage:
        enabled: true
        # Size of the PVC created
        size: 20Gi
        storageClass: gp2
        mountPath: "/vault/data"
      prometheusRules:
        enabled: true

Other useful info:

Pod - vault-0

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  98s   default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

PVC - data-vault-0

kubectl describe pvc data-vault-0 -n vault
Name:          data-vault-0
Namespace:     vault
StorageClass:  gp2
Status:        Pending
Volume:        
Labels:        app.kubernetes.io/instance=vault
               app.kubernetes.io/name=vault
               component=server
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: redacted
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       vault-0
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForFirstConsumer  10m                   persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  ExternalProvisioning  4m14s (x25 over 10m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

PV

 kubectl get pv -n vault
No resources found

Expected behavior
Vault PV being provisioned during the Helm deployment. Resulting in the Vault pod vault-0 not being stuck in pending status and being accessible.

Environment
EKS - 1.25

vault-helm version:
version: "0.25.0"

Chart values:

values:
    server:
      ingress:
        enabled: true
        hosts:
          - host: hostname.com
      dataStorage:
        enabled: true
        # Size of the PVC created
        size: 20Gi
        storageClass: gp2
        mountPath: "/vault/data"
      prometheusRules:
        enabled: true

[AlyRagab]

Same exact issue with me running Kubernetes 1.27 and latest vault-helm version.

[BJWRD]

Update:
This is not an issue connected to the Vault Helm Chart.

An ebs csi controller is required to be installed upon the cluster prior to the Vault deployment. This resolves the scheduling issues.

[rajeshchella2887]

can you please share detailed steps to fix this issue..

i am using k8s version 1.28 and trying to install vault using helm chart..

NAME READY STATUS RESTARTS AGE
vault-prod-0 0/1 Pending 0 25m
vault-prod-1 0/1 Pending 0 25m
vault-prod-2 0/1 Pending 0 25m
vault-prod-3 0/1 Pending 0 25m
vault-prod-4 0/1 Pending 0 25m
vault-prod-agent-injector-c7b4d46b8-hc8xl 1/1 Running 0 25m

kube@k8s-master1:/opt/vault/vault-production$ kubectl describe pvc data-vault-prod-0 -n vault
Name: data-vault-prod-0
Namespace: vault
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/instance=vault-prod
app.kubernetes.io/name=vault
component=server
Annotations:
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: vault-prod-0
Events:
Type Reason Age From Message

---

Normal FailedBinding 14m (x62 over 30m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Normal FailedBinding 3m57s (x26 over 10m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
kube@k8s-master1:/opt/vault/vault-production$

[dev-gto]

This error also occurs on microk8s. It works on minikube, though.

[helenzhou0523]

Is there any solution to this?

[tsahui]

I had the exact same issue.
so for all you other guys asking for detailed steps...

you need to:

1. attach the IAM role managing the cluster with AmazonEBSCSIDriverPolicy
2. when you create the cluster you need to add Amazon EBS CSI driver add-on (https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html)
3. make sure the cluster's StorageClass resource is annotated with storageclass.kubernetes.io/is-default-class=true (or create a new 1 with the annotation if you need a different one

then you can install vault or any other app that requires persistent volumes (the above is for using EBS, but is true for EFS or other data storages as well)