Pebble Helm chart - Let's Encrypt for unreachable CI environment with Kubernetes!

WARNING: Pebble as an ACME server and this Helm chart is only meant for testing purposes, it is not secure and not meant for production.

Pebble is an ACME server like Let's Encrypt. ACME servers can provide TLS certificates for HTTP over TLS (HTTPS) to ACME clients that are able to prove control over a domain name through an ACME challenge.

This Helm chart makes it easy to install Pebble in a Kubernetes cluster using Helm and work with fake domain names.

Motivation

This Helm chart is meant to help test applications deployments against an ACME server like Let's Encrypt from a unreachable CI environments, because using Let's Encrypts staging environment likely wouldn't work there.

In the commonly used HTTP-01 ACME challenge, an ACME client need to prove its control of a domain's web server. During this challenge, the ACME server will first independently lookup the domain name's IP, and make a web request to it, and that's the problem! The domain name needs to be publicly recognized as well as accessible, both these parts are problematic while working with an ephemeral CI environment.

Overview

This Helm chart deploys an ACME server (Pebble) and a DNS server (CoreDNS) that the ACME server will rely on for domain lookups, which you can configure as you please (Corefile).

Installation

# Note that pebble is written twice below, the first is the helm chart name as
# found in the Helm chart repository while the second is the name of the
# "helm release" which is the Helm official terminology for an installation
# of a Helm chart.
helm install pebble pebble --repo=https://jupyterhub.github.io/helm-chart/

Chart configuration

To configure the Helm chart, create my-values.yaml to pass with the --values flag during helm install or helm upgrade.

Helm charts render templates into Kubernetes yaml files using configurable values. Helm charts contain default values, and these can be overridden in a merging process during chart installation and upgrades, for example with the --values flag to pass a YAML file with values or with the --set flag to override specific keys' values.

CoreDNS configuration options

Pebble as an ACME server during ACME challenges will make DNS lookups. This Helm chart makes those lookups go through a dedicated CoreDNS DNS server that you can configure to manipulate what IP address various lookups should resolve to. Without additional configuration, it will just reference the Kubernetes cluster's DNS server.

CoreDNS configuration is documented here, and the current configuration can be inspected like this.

# Pebble's current configuration of CoreDNS
kubectl get configmap --all-namespaces -l app.kubernetes.io/name=pebble-coredns -o jsonpath='{.items[0].data.Corefile}'

You can inject a section in this configuration through Pebble's Helm chart configuration, for example like this.

# Pebble's Helm chart configuration (my-values.yaml)
coredns:
  # make all DNS lookups to "test" and subdomains go to the
  # Kubernetes service named mysvc in the same namespaces as
  # Pebble is installed
  corefileSegment: |-
    template ANY ANY test {
      answer "{{ .Name }} 60 IN CNAME mysvc.{$PEBBLE_NAMESPACE}.svc.cluster.local"
    }

Notes about the example above

1. {$ENV_VAR_NAME} is the syntax to reference environment variables.
2. PEBBLE_NAMESPACE is an environment variable explicitly on the CoreDNS pod to reference the namespace where Pebble is installed.
3. We provide a CNAME record that acts like an alias domain name, which IP is in turn looked up by CoreDNS with the Kubernetes cluster's DNS server.
4. The CNAME must reference the full domain name, not only mysvc for example.

Referencing the CoreDNS server

You may want other lookups to also go to this DNS server. It is available through this Kubernetes service.

# Pebble's CoreDNS Kubernetes service
kubectl get svc --all-namespaces -l app.kubernetes.io/name=pebble-coredns

You then have many options which can't be covered here, but here are some noteworthy pieces to be aware about.

1. Kubernetes Service's environment variables
2. Kubernetes Pod's DNS config
3. Kubernetes clusters own DNS server's configuration

Pebble configuration options

Custom challenge ports During a HTTP-01 (80) and TLS-ALPN-01 (443) challenge, Pebble will connect to the domains IP address using specific ports, but since it is meant for testing, you can configure these ports. This is useful if your ACME client is behind a Kubernetes service exposing it on port 8080 for example.

pebble:
  config:
    pebble:
      httpPort: 80 # this is the port where outgoing HTTP-01 challenges go
      tlsPort: 443 # this is the port where outgoing TLS-ALPN-01 challenges go

Mischievous behavior?

Pebble is developed to test ACME clients and ensure they are robust, so it can intentionally act mischievous.

The default of this Helm chart seen below configures Pebble to ensure speedy certificate acquisition. Note that if you provide an array to pebble.env, it will override the default array of environment variables.

pebble:
  env:
    ## ref: https://github.com/letsencrypt/pebble#testing-at-full-speed
    - name: PEBBLE_VA_NOSLEEP
      value: "1"
    ## ref: https://github.com/letsencrypt/pebble#invalid-anti-replay-nonce-errors
    - name: PEBBLE_WFE_NONCEREJECT
      value: "0"
    ## ref: https://github.com/letsencrypt/pebble#authorization-reuse
    - name: PEBBLE_AUTHZREUSE
      value: "100"

See Pebble's documentation for more info about its mischievous behavior.

ACME Client configuration

The ACME client needs configuration about where Pebble as an ACME server is and that it should accept the root TLS certificates signing the leaf TLS certificates used for Pebble's own HTTPS communication.

1. Provide an URL

Typically you should provide https://pebble/dir, but generally https://pebbles-service-name.pebbles-namespace/dir if Pebble is renamed or your ACME client is in another Kubernetes namespace.

PEBBLE_SERVICE_NAME=$(kubectl get svc --all-namespaces -l app.kubernetes.io/name=pebble -o jsonpath='{.items[0].metadata.name}')
PEBBLE_NAMESPACE=$(kubectl get svc --all-namespaces -l app.kubernetes.io/name=pebble -o jsonpath='{.items[0].metadata.namespace}')
echo https://${PEBBLE_SERVICE_NAME}.${PEBBLE_NAMESPACE}/dir

2. Accept a certificate

WARNING: Never trust a root certificate with exposed key, you may think the communication is secure!

Communication with Pebble's ACME server and its management REST API requires accepting a certificate Pebble use. Pebble will use a leaf certificate dynamically created with this unsafe root certificate and key.

Watch out for confusion!

There are two different root certificates to discuss!

One root certificate is associated with Pebble's own communication, and the other is ephemeral and what Pebble use to create certificates for its ACME clients completing ACME challenges. Pebble recreates the ephemeral root certificate on startup and exposes it and its associated key through the management REST API on https://pebble:8444/roots/0.

Example ACME client configuration

The ACME client will need to be configured to either accept all certificates or accept the root certificate to trust and be configured to trust it. In this example we will assume we explicitly provide the certificate to trust, and that we use the LEGO ACME client.

A Kubernetes ConfigMap for the root certificate

We are going to put the root certificate for the ACME client to trust within a Kubernetes ConfigMap, declare it as a volume in the ACME client's pod, and then mount it as a file in the pod's container.

If the ACME client is in the Kubernetes namespace where Pebble was installed, the ACME client can reference an existing ConfigMap that contains the root certificate it needs to trust. The ConfigMap can be identified like this.

# the configmap with the root certificate the ACME client need to trust
kubectl get configmap --all-namespaces -l app.kubernetes.io/name=pebble

If the ACME client was in a different namespace, you can create create a ConfigMap in its namespace like this.

cat <<EOF | kubectl apply --namespace <namespace-of-acme-client> -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: pebble
data:
  root-cert.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDSzCCAjOgAwIBAgIIOvR7X+wFgKkwDQYJKoZIhvcNAQELBQAwIDEeMBwGA1UE
    AxMVbWluaWNhIHJvb3QgY2EgM2FmNDdiMCAXDTIwMDQyNjIzMDYxNloYDzIxMjAw
    NDI3MDAwNjE2WjAgMR4wHAYDVQQDExVtaW5pY2Egcm9vdCBjYSAzYWY0N2IwggEi
    MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaoISLUOImo7vm7sGUpeycouDP
    TcJj6CxfCbvBsrlAg8ERGIph9H7TuDnTVk46pOaoxByGlwvvh4qR/Dled+G8NCt5
    s0r0yemY/fx1grm1TmcJRO+A1P5kx/M9hy+kVcyLRvPOnvo8Thj/4zvaJDh+pSjt
    5oAQvOHt9hYwGkkvSsZw12cTUuCsbypQ4lapDSeAjp3pNlqFcWmCvF9Ib3URDybN
    JWhY6yQQe54D2LxYqxCfYZjKhNbaxlNTlHu0Ujy75I/AdSjK6DljAZh0OimuQNEm
    FyXWvpnfyHbV5f0mMiXIOo2FY8izSD7cyFagmr0XvymCtxeDK1+MvT2pM+rXAgMB
    AAGjgYYwgYMwDgYDVR0PAQH/BAQDAgKEMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
    BgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBR0MgecNe4RY575
    qAtIt6zAjbBqLTAfBgNVHSMEGDAWgBR0MgecNe4RY575qAtIt6zAjbBqLTANBgkq
    hkiG9w0BAQsFAAOCAQEAjtGjoXGRG7586vyT3XcJBa8y9MOsDhQGOec23h40NJCn
    SPF28bmTIaWhB+Hv8G+Mkyf9Ov3L5L/mH0VGvZUkMAnSdT4vaMYGrTvMtYGS/8ew
    lPnlSJ3oO9Kz9zfOneoPDD1OGkV0Oq3wLn9cq6jQgItEeACsXNtaogXJxYhvxiV1
    1k/gjXmG9pvFpb0A1bw6apxGftIViDKrPR2P/pG3QAuLKywQiNxZ5odf3kvKdZmJ
    hLbu119My9XiiWhNegufcRNRNEnKJ5AQsBEwLEnD4oeIZmFvYVKOPjfWRV5qczVi
    mUPjtQv88HhlgX/lBVWJ2VONlFWVoOreZz4GkAm5bA==
    -----END CERTIFICATE-----
EOF

Mounting the ConfigMap

With a ConfigMap available in the ACME client's namespace, presumable named pebble and having a key named root-cert.pem, we can now mount it as a file in a pod where the ACME client runs.

# ... within a Pod specification where the ACME client runs
volumes:
  - name: pebble-configmap
    configMap:
      name: pebble # name of configmap
containers:
  - name: my-container-with-an-acme-client
    # ...
    volumeMounts:
      - name: pebble-configmap # name of volume
        subPath: root-cert.pem # name of specific key in configmap
        mountPath: /etc/pebble/root-cert.pem # name of file to mount

Referencing the certificate file

In this example where we presume the LEGO ACME client, we configure it to trust a root certificate and its signed leaf certificates through the LEGO_CA_CERTIFICATES environment variable.

# ... within a Pod specification where a LEGO ACME client runs
containers:
  - name: my-container-with-a-lego-acme-client
    # ...
    env:
      - name: LEGO_CA_CERTIFICATES
        value: /etc/pebble/root-cert.pem # reference the mounted file!

Local development

Prerequisites

• docker
• k3d
• kubectl
• helm

Setup

# clone the git repo
git clone https://github.com/jupyterhub/pebble-helm-chart.git
cd pebble-helm-chart

# setup a local k8s cluster
k3d create --wait 60 --publish 8443:32443 --publish 8444:32444 --publish 8053:32053/udp --publish 8053:32053/tcp
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"

# install pebble
helm upgrade --install pebble ./pebble --cleanup-on-fail

Test

# run a basic health check
helm test pebble

kubectl logs pebble-test --all-containers --prefix
kubectl logs pebble-coredns-test --all-containers --prefix
kubectl logs deploy/pebble --all-containers --prefix
kubectl logs deploy/pebble-coredns --all-containers --prefix

Release

No changelog or similar yet. Making a release is as easy as pushing a tagged commit on the main branch.

git tag -a x.y.z -m x.y.z
git push --follow-tags