Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raising replica and min_quorom to 6 does not allow node to auto join raft into ready state #894

Open
nia-potato opened this issue May 18, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@nia-potato
Copy link

Describe the bug
after raising replica and min_quorom to 6 in the helm chart, running k get pods only shows 5 pods are ready instead of joining the 6 pod to raft.

To Reproduce
Steps to reproduce the behavior:
install chart with config of

  ha:
    enabled: true
    replicas: 6
    raft:
      enabled: true
      setNodeId: true
      config: |
*********OTHER_CONFIG******
 storage "raft" {
          path = "/vault/data"
          performance_multiplier = 1
          retry_join {
            auto_join = "provider=k8s label_selector=\"app=vault,component=server\" namespace=\"vault\" "
            leader_tls_servername = "primary-vault"
            auto_join_scheme = "https"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/vault-ca.pem"
            leader_client_cert_file = "/vault/userconfig/vault-tls/vault-cert.pem"
            leader_client_key_file   = "/vault/userconfig/vault-tls/vault-key.pem"
          }
          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 6
            server_stabilization_time = "10s"
          }
        }
k get pods 
primary-vault-0   1/1     Running   0          117s
primary-vault-1   1/1     Running   0          117s
primary-vault-2   1/1     Running   0          117s
primary-vault-3   1/1     Running   0          117s
primary-vault-4   1/1     Running   0          117s
primary-vault-5   0/1     Running   0          117s

Other useful info to include: vault pod logs, kubectl describe statefulset vault and kubectl get statefulset vault -o yaml output

Expected behavior
replica of 6 + min_quorum of 6 should have 6 nodes all in ready state and joined in raft.

Environment

  • Kubernetes version: GKE, vault-enterprise
@nia-potato nia-potato added the bug Something isn't working label May 18, 2023
@maxb
Copy link

maxb commented Jun 7, 2023

  • You have not mentioned which seal method is in use for this Vault. A new node requires access to the unseal key to join the cluster, which may mean manual unsealing.

  • A Raft cluster of 6 nodes is questionable, since it is no more fault-tolerant than a 5 node cluster. The failure tolerance is 2 in both cases.

  • I'm pretty sure the autopilot settings in the configuration file only apply when initializing a new cluster, and to modify an existing cluster, changes must be made via the Vault API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants