Cluster Mode

Remote Signer Cluster Mode

Remote Signer supports a cluster mode when running inside kubernetes that enables secure password sharing between all nodes, in a way that even if the private key is unlocked in a specific node, it will be available for other trusted nodes. To see more details how to configure Remote Signer as a pod, check Running in Kubernetes

The node trust source is a common GPG Key that is shared between the nodes. The GPG key is specified by the following environment variables:

• MASTER_GPG_KEY_PATH => Path of the GPG Key File (the file should be the same for all nodes)
• MASTER_GPG_KEY_PASSWORD_PATH => Path of the GPG Key Password File
• MASTER_GPG_KEY_BASE64_ENCODED => If the GPG Key is base64 encoded

This GPG key should be set up in a strict secret inside the kubernetes cluster together with the password. The Remote-Signer uses Kubernetes API to discover other remote-signer nodes using the same namespace. To enable the Remote Signer to discover other nodes, a RBAC rule should be created allowing the nodes to check the namespace:

kubectl create rolebinding pod-reader --clusterrole=view --serviceaccount=PODNAME:default --namespace=POD_NAMESPACE

So for example if the remote-signer pod is named server and is inside a remote-signer namespace you should run:

kubectl create rolebinding pod-reader --clusterrole=view --serviceaccount=server:default --namespace=remote-signer

Then they should be able to discover themselves and other nodes:

...
INFO| RemoteSigner   | Remote Signer is now listening at 0.0.0.0:5100
INFO| Kubernetes     | Starting Kubernetes Routine
INFO| Kubernetes     | Kubernetes Namespace: remote-signer
INFO| Kubernetes     | Pod Hostname: server-5bb7f59794-jcphc
INFO| Kubernetes     | Pod ID: 4bea628e-571c-11e9-a7ec-468ba104692f
INFO| Kubernetes     | To avoid concurrency on cluster starting we're waiting 1 second plus some random time
INFO| Kubernetes     | The exact time is 4000 ms
INFO| Kubernetes     | Checking for other remote-signer nodes...
INFO| Kubernetes     | There are 3 pods (including me). Fetching encrypted passwords...
INFO| Kubernetes     | Received 6 passwords from 10.42.15.178
INFO| GPG Endpoint   | [200] ( 0.20 ms) { 2 bytes} POST /remoteSigner/__internal/__postEncryptedPasswords from ::1
INFO| Kubernetes     | Received 6 passwords from 10.42.18.73
INFO| GPG Endpoint   | [200] ( 0.15 ms) { 2 bytes} POST /remoteSigner/__internal/__postEncryptedPasswords from ::1
INFO| Kubernetes     | Received 12 passwords from 3 pods. Triggering Local Unlock
INFO| SecretsManager | Unlocking key 6EFF0C82BE03FFD9
...

Distributed GPG public key store

For distributing the GPG Public Key store (which is used in validations) you should use a database connection for that and or a caching layer.

The available databases:

• postgres
  • ENABLE_DATABASE => true
  • DATABASE_DIALECT => postgres
  • CONNECTION_STRING => postgres://postgres:123456@localhost:5432/postgres?sslmode=disable&search_path=remote_signer
  • DATABASE_TOKEN_MANAGER => true (needs a caching layer)
  • DATABASE_AUTH_MANAGER => true (needs a caching layer)
• rethinkdb DEPRECATED
  • ENABLE_DATABASE => true
  • DATABASE_DIALECT => postgres
  • RETHINKDB_HOST => localhost
  • RETHINKDB_PORT => 28015
  • RETHINKDB_USERNAME => admin
  • RETHINKDB_PASSWORD => ``
  • DATABASE_TOKEN_MANAGER => true
  • DATABASE_AUTH_MANAGER => true

The available caching layers:

• redis
  • REDIS_ENABLE => true
  • REDIS_TLS_ENABLED => true if needs to use TLS auth, false otherwise
  • REDIS_HOST => localhost:6379
  • REDIS_USER => If need auth, this is username. Empty otherwise
  • REDIS_PASS => If need auth, this is password. Empty otherwise
  • REDIS_MAX_LOCAL_TTL => Local in-memory cache (inside remote signer) max Time to live. 5m
  • REDIS_MAX_LOCAL_OBJECTS => Local in-memory cache (inside remote signer) max number of objects. 100
  • REDIS_CLUSTER_MODE => true if the redis instance is running cluster mode (e.g. AWS ElasticCache). false otherwise.

Other functions

Besides these functions, it might be good to take a look into these features:

• Vault Private Key Backend
• Running in Kubernetes