Get smokescreen up to date with the upstream (#2)

* add a custom interface for the resolver instead of forcing *net.Resolver (stripe#187) * feature/add prometheus metrics (stripe#179) * STORY-25143 - Add prometheus metrics to smokescreen * STORY-25143 - Cleanup * STORY-25143 - Fix tests to compare new metric labels * STORY-25143 - Host prometheus endpoint on separate port * STORY-25143 - Use value provided via command line flag * STORY-25143 - Add prometheus timing metrics * STORY-25143 - Fix nil map assignment and prometheus metric name sanitisation * STORY-25143 - Cleanup comments * STORY-25143 - Remove some repetition + add further unit testing * STORY-25143 - Document new prometheus features in README + add port flag to prometheus config * STORY-25143 - Make PR requested changes: * Don't export metrics list * Follow project sytlistic choices * STORY-25143 - Rename only one receiver * STORY-25143 - Add new `--expose-prometheus-metrics` flag to CLI to toggle exposing prometheus metrics * Small cleanup of timer metrics * Fix go module vendoring * Use ElementsMatch to ignore order * Just use require * Move the custom request handler call after the main acl check * Use local server instead of httpbin (stripe#192) * Do not return a denyError for DNS resolution failures (stripe#194) * dont return denial errors for dns resolution failures * fix test * move DNSError check into net.Error assertion, extend test * fix integration test * add AcceptResponseHandler to modify accepted responses (stripe#196) * add AcceptResponseHandler to modify accepted responses * customer->custom * Update docs to clarify global_deny_list (stripe#197) * update docs to clarify global_deny_list behavior * consistent example domain * be more concise * Use AcceptResponseHandler in goproxy https CONNECT hook (stripe#199) * pipe AcceptResponseHandler into new goproxy hook * update comment * go mod vendor * unit test * use smokescreenctx in acceptresponsehandler * fix unit tests * Export SmokescreenContext type (stripe#200) * export SmokescreenContext type * also export AclDecision * ResolvedAddr too * consistent caps * Update pkg/smokescreen/smokescreen.go Co-authored-by: jjiang-stripe <55402658+jjiang-stripe@users.noreply.github.com> * export Decision --------- Co-authored-by: jjiang-stripe <55402658+jjiang-stripe@users.noreply.github.com> * generate new test pki (stripe#206) * allow listen address specification for prom (stripe#203) * Bump golang.org/x/net from 0.7.0 to 0.17.0 (stripe#204) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.17.0. - [Commits](golang/net@v0.7.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bump go versions (stripe#207) * update dependency * configure addr in smokescreen and add unit test * use fmt * try this workaround * variable name change * Update docs to disambiguate ACL vs --deny-address behavior (stripe#210) * update docs to clarify how IP filtering works --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: JulesD <JulesDT@users.noreply.github.com> Co-authored-by: Josh McConnell <josh.mcconnell465@gmail.com> Co-authored-by: Kevin Vincent <kevinv@stripe.com> Co-authored-by: kevinv-stripe <102822342+kevinv-stripe@users.noreply.github.com> Co-authored-by: Sergey Rud <sergeyrud@stripe.com> Co-authored-by: cmoresco-stripe <106690468+cmoresco-stripe@users.noreply.github.com> Co-authored-by: Craig Shannon <cds@stripe.com> Co-authored-by: jjiang-stripe <55402658+jjiang-stripe@users.noreply.github.com> Co-authored-by: Timofey Bakunin <36561672+ne-bknn@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yuxi Xie <xieyuxi@stripe.com> Co-authored-by: xieyuxi-stripe <141708814+xieyuxi-stripe@users.noreply.github.com>
intercom · Jan 4, 2024 · f2e36a3 · f2e36a3
1 parent 83ed067
commit f2e36a3
Show file tree

Hide file tree

Showing 617 changed files with 108,730 additions and 5,471 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -1,10 +1,14 @@
 on: [push, pull_request]
 name: Test
+# Workaround for coveralls error "Can't add a job to a build that is already closed"
+# See https://github.com/lemurheavy/coveralls-public/issues/1716
+env:
+  COVERALLS_SERVICE_NUMBER: ${{ github.run_id }}-${{ github.run_attempt }}
 jobs:
   test:
     strategy:
       matrix:
-        go-version: ['1.19', '1.20']
+        go-version: ['1.20', '1.21']
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
@@ -23,6 +27,7 @@ jobs:
         go test -race -v -timeout 2m -failfast -covermode atomic -coverprofile=.covprofile ./... -tags=nointegration
         # Run integration tests hermetically to avoid nondeterministic races on environment variables
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestSmokescreenIntegration
+        go test -race -v -timeout 2m -failfast ./cmd/... -run TestInvalidUpstreamProxyConfiguratedFromEnv
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestInvalidUpstreamProxyConfiguration
         go test -race -v -timeout 2m -failfast ./cmd/... -run TestClientHalfCloseConnection
     - name: Install goveralls

diff --git a/README.md b/README.md
@@ -3,10 +3,13 @@
 Smokescreen is a HTTP CONNECT proxy. It proxies most traffic from Stripe to the
 external world (e.g., webhooks).
 
-Smokescreen restricts which URLs it connects to: it resolves each domain name
-that is requested and ensures that it is a publicly routable IP and not a
-Stripe-internal IP. This prevents a class of attacks where, for instance, our
-own webhooks infrastructure is used to scan Stripe's internal network.
+Smokescreen restricts which URLs it connects to:
+- It uses a pre-configured hostname ACL to only allow requests addressed to certain allow-listed hostnames, 
+to ensure that no malicious code is attempting to make requests to unexpected services.
+- It also resolves each domain name that is requested, and ensures that it is a publicly routable 
+IP address and not an internal IP address. This prevents a class of attacks where, for instance, 
+our own webhooks infrastructure is used to scan Stripe’s internal network. Smokescreen 
+can also be further configured to allow or deny specific IP addresses or ranges.
 
 Smokescreen also allows us to centralize egress from Stripe, allowing us to give
 financial partners stable egress IP addresses and abstracting away the details
@@ -58,6 +61,11 @@ Here are the options you can give Smokescreen:
    --deny-address value                        Add IP[:PORT] to list of blocked IPs.  Repeatable.
    --allow-address value                       Add IP[:PORT] to list of allowed IPs.  Repeatable.
    --egress-acl-file FILE                      Validate egress traffic against FILE
+   --expose-prometheus-metrics                 Exposes metrics via a Prometheus scrapable endpoint.
+   --prometheus-endpoint ENDPOINT              Specify endpoint to host Prometheus metrics on. (default: "/metrics")
+                                                 Requires `--expose-prometheus-metrics` to be set.
+   --prometheus-port PORT                      Specify port to host Prometheus metrics on. (default "9810")
+                                                 Requires `--expose-prometheus-metrics` to be set.
    --resolver-address ADDRESS                  Make DNS requests to ADDRESS (IP:port).  Repeatable.
    --statsd-address ADDRESS                    Send metrics to statsd at ADDRESS (IP:port). (default: "127.0.0.1:8200")
    --tls-server-bundle-file FILE               Authenticate to clients using key and certs from FILE
@@ -112,10 +120,13 @@ func main() {
 	smokescreen.StartWithConfig(conf, nil)
 }
 ```
+### IP Filtering
 
-### ACLs
+To control the routing of requests to specific IP addresses or IP blocks, use the `deny-address`, `allow-address`, `deny-range`, and `allow-range` options in the config. 
 
-An ACL can be described in a YAML formatted file. The ACL, at its top-level, contains a list of services as well as a default behavior.
+### Hostname ACLs
+
+A hostname ACL can be described in a YAML formatted file. The ACL, at its top-level, contains a list of services as well as a default behavior.
 
 Three policies are supported:
 
@@ -125,6 +136,8 @@ Three policies are supported:
 | Report  | Allows all traffic for this service and warns if client accesses a remote host which is not in the list        |
 | Enforce | Only allows traffic to remote hosts provided in the list. Will warn and deny if remote host is not in the list |
 
+> :warning: **The ACL is only applied to hostnames *as they appear in the request*!** If you want to allow or deny traffic based on the destination IP address *after DNS resolution*, you should be using the config options instead (see the `IP Filtering` section above).
+
 A host can be specified with or without a globbing prefix. The host (without the globbing prefix) must be in Punycode to prevent ambiguity.
 
 | host                | valid   |
@@ -139,9 +152,9 @@ A host can be specified with or without a globbing prefix. The host (without the
 
 [Here](https://github.com/stripe/smokescreen/blob/master/pkg/smokescreen/acl/v1/testdata/sample_config.yaml) is a sample ACL.
 
-#### Global Allow/Deny Lists
+#### Global Hostname Allow/Deny Lists
 
-Optionally, you may specify a global allow list and a global deny list in your ACL config.
+Optionally, you may specify a global allow list and a global deny list for hostnames in your ACL config.
 
 These lists override the policy, but do not override the `allowed_domains` list for each role.
 
@@ -150,6 +163,8 @@ For example, specifying `example.com` in your global_allow_list will allow traff
 Similarly, specifying `malicious.com` in your global_deny_list will deny traffic for that domain on a role, even if that role is set to `report` or `open`.
 However, if the host specifies `malicious.com` in its `allowed_domains`, traffic to `malicious.com` will be allowed on that role, regardless of policy.
 
+> :warning: **The global_deny_list will only block specific *hostnames*, not entire *destinations*.** For example, if `malicious.com` is in the global_deny_list but the IP address that it resolves to is not, roles with an `open` policy will still be able to access the destination by using its IP address directly. For this reason, **we recommend using allowlists instead of denylists** whenever it is possible to do so, and **blocking IP addresses via config options, not the ACL** (see the `IP Filtering` section above).
+
 If a domain matches both the `global_allow_list` and the `global_deny_list`, the `global_deny_list` behavior takes priority.
 
 [Here](https://github.com/stripe/smokescreen/blob/master/pkg/smokescreen/acl/v1/testdata/sample_config_with_global.yaml) is a sample ACL specifying these options.

diff --git a/cmd/integration_test.go b/cmd/integration_test.go
@@ -279,7 +279,7 @@ func TestSmokescreenIntegration(t *testing.T) {
 
 	for _, useTLS := range []bool{true, false} {
 		// Smokescreen instances
-		_, proxyServer, err := startSmokescreen(t, useTLS, &logHook)
+		_, proxyServer, err := startSmokescreen(t, useTLS, &logHook, "")
 		require.NoError(t, err)
 		defer proxyServer.Close()
 		proxyServers[useTLS] = proxyServer
@@ -441,21 +441,21 @@ func validateProxyResponseWithUpstream(t *testing.T, test *TestCase, resp *http.
 	t.Logf("HTTP Response: %#v", resp)
 
 	if test.OverConnect {
-		a.Contains(err.Error(), "Failed to connect to remote host")
+		a.Contains(err.Error(), "Failed to resolve remote hostname")
 	} else {
 		a.Equal(http.StatusBadGateway, resp.StatusCode)
 	}
 }
 
 // This test must be run with a separate test command as the environment variables
 // required can race with the test above.
-func TestInvalidUpstreamProxyConfiguration(t *testing.T) {
+func TestInvalidUpstreamProxyConfiguratedFromEnv(t *testing.T) {
 	var logHook logrustest.Hook
 	servers := map[bool]*httptest.Server{}
 
 	// Create TLS and non-TLS instances of Smokescreen
 	for _, useTLS := range []bool{true, false} {
-		_, server, err := startSmokescreen(t, useTLS, &logHook)
+		_, server, err := startSmokescreen(t, useTLS, &logHook, "")
 		require.NoError(t, err)
 		defer server.Close()
 		servers[useTLS] = server
@@ -500,6 +500,58 @@ func TestInvalidUpstreamProxyConfiguration(t *testing.T) {
 	}
 }
 
+func TestInvalidUpstreamProxyConfiguration(t *testing.T) {
+	var logHook logrustest.Hook
+	servers := map[bool]*httptest.Server{}
+
+	// Create TLS and non-TLS instances of Smokescreen
+	for _, useTLS := range []bool{true, false} {
+		var httpProxyAddr string
+		if useTLS {
+			httpProxyAddr = "https://notaproxy.prxy.svc:443"
+		} else {
+			httpProxyAddr = "http://notaproxy.prxy.svc:80"
+		}
+		_, server, err := startSmokescreen(t, useTLS, &logHook, httpProxyAddr)
+		require.NoError(t, err)
+		defer server.Close()
+		servers[useTLS] = server
+	}
+
+	// Passing an illegal upstream proxy value is not designed to be an especially well
+	// handled error so it would fail many of the checks in our other tests. We really
+	// only care to ensure that these requests never succeed.
+	for _, overConnect := range []bool{true, false} {
+		t.Run(fmt.Sprintf("illegal proxy with CONNECT %t", overConnect), func(t *testing.T) {
+			var proxyTarget string
+			var upstreamProxy string
+
+			// These proxy targets don't actually matter as the requests won't be sent.
+			// because the resolution of the upstream proxy will fail.
+			if overConnect {
+				upstreamProxy = "https://notaproxy.prxy.svc:443"
+				proxyTarget = "https://api.stripe.com:443"
+			} else {
+				upstreamProxy = "http://notaproxy.prxy.svc:80"
+				proxyTarget = "http://checkip.amazonaws.com:80"
+			}
+
+			testCase := &TestCase{
+				OverConnect:   overConnect,
+				OverTLS:       overConnect,
+				ProxyURL:      servers[overConnect].URL,
+				TargetURL:     proxyTarget,
+				UpstreamProxy: upstreamProxy,
+				RoleName:      generateRoleForPolicy(acl.Open),
+				ExpectStatus:  http.StatusBadGateway,
+			}
+			resp, err := executeRequestForTest(t, testCase, &logHook)
+			validateProxyResponseWithUpstream(t, testCase, resp, err, logHook.AllEntries())
+
+		})
+	}
+}
+
 func TestClientHalfCloseConnection(t *testing.T) {
 	a := assert.New(t)
 
@@ -510,7 +562,7 @@ func TestClientHalfCloseConnection(t *testing.T) {
 
 	var logHook logrustest.Hook
 
-	conf, server, err := startSmokescreen(t, false, &logHook)
+	conf, server, err := startSmokescreen(t, false, &logHook, "")
 	require.NoError(t, err)
 	defer server.Close()
 
@@ -577,7 +629,7 @@ func findLogEntry(entries []*logrus.Entry, msg string) *logrus.Entry {
 	return nil
 }
 
-func startSmokescreen(t *testing.T, useTLS bool, logHook logrus.Hook) (*smokescreen.Config, *httptest.Server, error) {
+func startSmokescreen(t *testing.T, useTLS bool, logHook logrus.Hook, httpProxyAddr string) (*smokescreen.Config, *httptest.Server, error) {
 	args := []string{
 		"smokescreen",
 		"--listen-ip=127.0.0.1",
@@ -596,6 +648,11 @@ func startSmokescreen(t *testing.T, useTLS bool, logHook logrus.Hook) (*smokescr
 		)
 	}
 
+	if httpProxyAddr != ""{
+		args = append(args, fmt.Sprintf("--upstream-http-proxy-addr=%s", httpProxyAddr))
+		args = append(args, fmt.Sprintf("--upstream-https-proxy-addr=%s", httpProxyAddr))
+	}
+
 	conf, err := NewConfiguration(args, nil)
 	if err != nil {
 		t.Fatalf("Failed to create configuration: %v", err)

diff --git a/cmd/smokescreen.go b/cmd/smokescreen.go
@@ -85,6 +85,25 @@ func NewConfiguration(args []string, logger *log.Logger) (*smokescreen.Config, e
 			Name:  "egress-acl-file",
 			Usage: "Validate egress traffic against `FILE`",
 		},
+		cli.BoolFlag{
+			Name:  "expose-prometheus-metrics",
+			Usage: "Expose metrics via prometheus.",
+		},
+		cli.StringFlag{
+			Name:  "prometheus-endpoint",
+			Value: "/metrics",
+			Usage: "Expose prometheus metrics on `ENDPOINT`. Requires --expose-prometheus-metrics to be set. Defaults to \"/metrics\"",
+		},
+		cli.StringFlag{
+			Name: "prometheus-listen-ip",
+			Value: "0.0.0.0",
+			Usage: "Listen for prometheus metrics on interface with address IP. Requires --expose-prometheus-metrics to be set. Defaults to \"0.0.0.0\"",
+		},
+		cli.StringFlag{
+			Name:  "prometheus-port",
+			Value: "9810",
+			Usage: "Expose prometheus metrics on `PORT`. Requires --expose-prometheus-metrics to be set. Defaults to \"9810\"",
+		},
 		cli.StringSliceFlag{
 			Name:  "resolver-address",
 			Usage: "Make DNS requests to `ADDRESS` (IP:port).  Repeatable.",
@@ -127,6 +146,16 @@ func NewConfiguration(args []string, logger *log.Logger) (*smokescreen.Config, e
 			Name:  "unsafe-allow-private-ranges",
 			Usage: "Allow private ip ranges by default",
 		},
+		cli.StringFlag{
+			Name:  "upstream-http-proxy-addr",
+			Value: "",
+			Usage: "Set Smokescreen's upstream HTTP proxy address",
+		},
+		cli.StringFlag{
+			Name:  "upstream-https-proxy-addr",
+			Value: "",
+			Usage: "Set Smokescreen's upstream HTTPS proxy address",
+		},
 	}
 
 	app.Action = func(c *cli.Context) error {
@@ -229,6 +258,12 @@ func NewConfiguration(args []string, logger *log.Logger) (*smokescreen.Config, e
 			}
 		}
 
+		if c.IsSet("expose-prometheus-metrics") {
+			if err := conf.SetupPrometheus(c.String("prometheus-endpoint"), c.String("prometheus-port"), c.String("prometheus-listen-ip")); err != nil {
+				return err
+			}
+		}
+
 		if c.IsSet("egress-acl-file") {
 			if err := conf.SetupEgressAcl(c.String("egress-acl-file")); err != nil {
 				return err
@@ -262,6 +297,14 @@ func NewConfiguration(args []string, logger *log.Logger) (*smokescreen.Config, e
 			}
 		}
 
+		if c.IsSet("upstream-http-proxy-addr") {
+			conf.UpstreamHttpProxyAddr = c.String("upstream-http-proxy-addr")
+		}
+
+		if c.IsSet("upstream-https-proxy-addr") {
+			conf.UpstreamHttpsProxyAddr = c.String("upstream-https-proxy-addr")
+		}
+
 		// Setup the connection tracker if there is not yet one in the config
 		if conf.ConnTracker == nil {
 			conf.ConnTracker = conntrack.NewTracker(conf.IdleTimeout, conf.MetricsClient, conf.Log, conf.ShuttingDown, nil)

diff --git a/cmd/testdata/pki/ca.pem b/cmd/testdata/pki/ca.pem
@@ -1,27 +1,33 @@
 -----BEGIN CERTIFICATE-----
-MIIEnjCCA4agAwIBAgIUVX3IM4foGZSavx8CYfWm5035gs8wDQYJKoZIhvcNAQEL
-BQAwgeYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
-Ew1TYW4gRnJhbmNpc2NvMSMwIQYDVQQKExpOb3QgQSBDb21wYW55IENBIEZvciBU
-ZXN0czFgMF4GA1UECxNXT29wcyBJIGRpZCBpdCBhZ2FpbiwgSSBwbGF5ZWQgd2l0
-aCB5b3VyIGhlYXJ0LCBnb3QgbG9zdCBpbiB0aGUgZ2FtZS4gT2ggYmFieSBiYWJ5
-IGJhYnkuMSMwIQYDVQQDExpUZXN0IENlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0x
-ODA4MTYyMDAwMDBaFw0yMzA4MTUyMDAwMDBaMIHmMQswCQYDVQQGEwJVUzETMBEG
-A1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEjMCEGA1UE
-ChMaTm90IEEgQ29tcGFueSBDQSBGb3IgVGVzdHMxYDBeBgNVBAsTV09vcHMgSSBk
-aWQgaXQgYWdhaW4sIEkgcGxheWVkIHdpdGggeW91ciBoZWFydCwgZ290IGxvc3Qg
-aW4gdGhlIGdhbWUuIE9oIGJhYnkgYmFieSBiYWJ5LjEjMCEGA1UEAxMaVGVzdCBD
-ZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
-AoIBAQDB/IYY40+YyUyo42bUh6JWgs5ywdCi7nokoYET4VnXiIvxtWAFksAZnNk2
-T+2RqXJuZbknZLNaOFhDsOz21q52zO57T3Q9q7X7XqeW7kAnQ5rkNV1liL3FbUgq
-pm9Ask3wRPU9qx5S8ToBdck25tN4mwjAHmqswaEhfGVK1rXJyuBb2CQO3AWADkOt
-Q8PbiEJus40elY49iYp7vRgNIt53HpD5L9O6hDGLCw0HQDPceGdAVc+CcJMlIJ/0
-W0rGmZbAEZVsoOYYQ0+2aPkIKw+PJ0DMXugZxlXFTUeHn55slMwL7nmN1MIYDElY
-YOvQR391npmjPuSGp9fEhgDmgjSTAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP
-BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSHjuAR3Zu6D8nvU8Uif7TEMN/ujjAN
-BgkqhkiG9w0BAQsFAAOCAQEAKBmSNB2Htl+nu1/PHrc5pi1UmV9ySrAi1g/DAzDR
-iGwDV5M+fUzdLg9Gb+Pg6b0HaIQRfUs4+qKgzbjCvqfdvk9806BH/1Rpvltiebom
-owk4XSr/28dEpx7VYBl/FC35J4NQ2IIiBM3m8vSoXHWxxONgq3kew0iPdeoJf7Sx
-h2UCaRlAd2IFjZcjUTqEY9nT6VzADoLpYuz5Fol41/iAsULnbut5ALI3IPFjzObm
-DapuCX9xWMoaZ9MzsPMZwxt/KF/NqBXwiOkqzdJhchcnmPerZGq7jIg7VV6pBr4j
-FGmMujqO/UZWu9D/QosIotF3OWlGL5NaZ8isVJ/GJrNJXg==
+MIIFtzCCA5+gAwIBAgIUcYbxViF/fKZg1BwwMGN2FL6EGTIwDQYJKoZIhvcNAQEL
+BQAwazELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcM
+DVNhbiBGcmFuY2lzY28xGTAXBgNVBAoMEFNtb2tlc2NyZWVuIFRlc3QxFDASBgNV
+BAsMC1Ntb2tlc2NyZWVuMB4XDTIzMTEwNzIyMDU1NVoXDTMzMTEwNDIyMDU1NVow
+azELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNh
+biBGcmFuY2lzY28xGTAXBgNVBAoMEFNtb2tlc2NyZWVuIFRlc3QxFDASBgNVBAsM
+C1Ntb2tlc2NyZWVuMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAx4a5
+2Y4XpdumUADBTa3oIY9kFHvH+yquzRyD4ngRCTKmDq4r1SKpZBQluG18ihf6f7GS
+7BFAl2wIcRt5fZtWmpKgZynLrCx+fiBVrnxKdsiVkZp/nY8hLDnweInYXX6J3UP2
+vmrTl+jGAoylDgQ3aS3gkhVc3QjI6pSg58cDKZYNX22J0whp2DiRHD5T1pYWWAaK
+5Aw0XR1SatvsLSKu6Xga7heTOZH0M06KYSs8h4BklSAdRR7xP+u1VqnXWPKetj8g
+WnqoEQummIrDEZMZAiV3KJANdkDfa66vKnEweIAymqempXK+35gqAmI3ErO4quWD
+mAaZ6iZLzpZxJRNpRyF7LNi5fntaOA67r80yDel6VfaDekoEqVwI6jXCSbLdqzyj
+lD47RQDVZmKX6SSi/skb9lsHc2HpejJyMEDXQPgn+keulIg+/088qWwGgRRTpipt
+NIl2+RSKnK+fqoIhUVQdV9yBJ3MVWXH/H2KfVrOy/TARsai9CsfC6Mt3xM5cOE0r
+WFjybZO1QStenRXjFKMD15zoVM+fYQRYTuO0t1ICSgWAbPNjWpYkTXocbYq/flGY
+KTGfRGAuiEMuRC1HSjgT+ZhARB353oNgfNinh8M/cwGpAPwvsUtLOSTI3z+gZ/eh
+0hE5zAUmI9a7X8t9cXDgiYb6PwTYt7CsY/0YTwMCAwEAAaNTMFEwHQYDVR0OBBYE
+FCJvjD/Gk7eKYbHd4RaBgeyQbY3SMB8GA1UdIwQYMBaAFCJvjD/Gk7eKYbHd4RaB
+geyQbY3SMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBAJ5FMe89
+sAQ0YtAj7LmBqx73DlrmwNZ/00N5IZJgoiBYxGQvsc7Y69vBSkiX0ZuJ5i1rkJqL
+ngJTdycOKzc1ofa6kf3JPrxk3EeGEEqCDdByI2eUB9oVxcGtqL1/eAXVYwFjXupY
+/CRkrv1qVSfioyNw1JrLxu0j+zqlfYFc/hUAvPm7HEHTnDKePxf+r1fqztudD4ld
+kjrcxW4otWHxEP4oyadyrcF7NWQK9vgzyXlB3lj57b8qlfrcniCnqYVFsRoCCFoa
++fETmhzgyNsUG5N2YfvDHd8yYuKyRKm+dDnVCN+RIov32oTvIFY5qHdpyjv30wvB
+0hhGpeTgeXY/ddVJu2BU8Ni8xgbKB9tRgPiO88bN4aWKoM6TRlhWfbcvpGKWY9ED
+CVzILzUDydhaqB8iC0NYlcBrooVl7MHkvl/7KW9QC+WKrsIXWeF48sUOtOpTvkfY
+nBjD+1RfHLML66WoferPNoDB1T6vkIcRKbHESYRfRwvsgD2fqVsepvJREt63yHKz
+KJHD/XxJAxMjrqAxqz1t4c/yg2dHA14jL/VgIooUfbvtiljFaDml0tdEIB4n0Ddy
+ckpj/iwIwJsg2yVVuStqWEPeErVmsdsDKbtn3KV6Y7wAQcSWFhJVYwiy78atgxOk
+CIQtLAJVo2DMgpZEHAmrkGlJ/a44blA/zd4b
 -----END CERTIFICATE-----
diff --git a/cmd/testdata/pki/crl.pem b/cmd/testdata/pki/crl.pem
@@ -1,21 +1,18 @@
 -----BEGIN X509 CRL-----
-MIIDijCCAnICAQEwDQYJKoZIhvcNAQELBQAwgeYxCzAJBgNVBAYTAlVTMRMwEQYD
-VQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMSMwIQYDVQQK
-ExpOb3QgQSBDb21wYW55IENBIEZvciBUZXN0czFgMF4GA1UECxNXT29wcyBJIGRp
-ZCBpdCBhZ2FpbiwgSSBwbGF5ZWQgd2l0aCB5b3VyIGhlYXJ0LCBnb3QgbG9zdCBp
-biB0aGUgZ2FtZS4gT2ggYmFieSBiYWJ5IGJhYnkuMSMwIQYDVQQDExpUZXN0IENl
-cnRpZmljYXRlIEF1dGhvcml0eRcNMTgwODE2MjAyOTE1WhcNMTkwMjEyMjAyOTE1
-WjAjMCECEHD0KS8QbJYZeDE3OU4loLoXDTE4MDgxNjIwMjkwNFqgggEwMIIBLDCC
-ASgGA1UdIwSCAR8wggEbgBSHjuAR3Zu6D8nvU8Uif7TEMN/ujqGB7KSB6TCB5jEL
-MAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBG
-cmFuY2lzY28xIzAhBgNVBAoTGk5vdCBBIENvbXBhbnkgQ0EgRm9yIFRlc3RzMWAw
-XgYDVQQLE1dPb3BzIEkgZGlkIGl0IGFnYWluLCBJIHBsYXllZCB3aXRoIHlvdXIg
-aGVhcnQsIGdvdCBsb3N0IGluIHRoZSBnYW1lLiBPaCBiYWJ5IGJhYnkgYmFieS4x
-IzAhBgNVBAMTGlRlc3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5ghRVfcgzh+gZlJq/
-HwJh9abnTfmCzzANBgkqhkiG9w0BAQsFAAOCAQEAL6CaIzWHsH1ndzG83OhqMsuI
-8Tlm8Yu8kYkCTvq4V5wpFUxtG7Pf8pKi1M8I9PsF+YTa58Atwb5RLMACi9RIj/hn
-zRMWMy4zUIChIuzM7y5MVT4z7amqjzRXZIlJOQkedjWW+qU3a2c5YYZjvDHUJ5dj
-MdAqU9lxafntKQDwOH8EYsMqSYNRQgJ6DzkRFb+ASaLvEtX+AOQhx55BsmpfEMNP
-dzDaQVvXIeGIWT3O9ZkHvIBS4+hUj1cWoCZncnAbNgUSW/LvxllucWQiyzYfAf9I
-J+djj/piXhXnU5AETozVFoIusSU3Fmh0FMfdHDl8y9EYGBZYMze1dluVlvmX5w==
+MIIC2zCBxAIBATANBgkqhkiG9w0BAQsFADBrMQswCQYDVQQGEwJVUzETMBEGA1UE
+CAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQ
+U21va2VzY3JlZW4gVGVzdDEUMBIGA1UECwwLU21va2VzY3JlZW4XDTIzMTEwNzIx
+NDMyMFoXDTIzMTIwNzIxNDMyMFowFDASAgEBFw0yMzExMDcyMTQzMTRaoA8wDTAL
+BgNVHRQEBAICEAEwDQYJKoZIhvcNAQELBQADggIBAI7yGPw4/A+7WMrqghIcsbZK
+UuLRA7R0AlKIzQGJ/4Tk+ZJF1mlhQrB7kB89yPq7b+CssldnvWpIH67yn3snKtdU
+QYOxvXfds2wJn4kyWo46Nnz4/1huc2r4sNEvyvTLEH6VRxivHGR2UncODPkNVpcu
+eBoFUwUZrh9aWUcBY1/ilPcWq7JwyGQsuIM7IBLkaWb75ZV+poEzufIVVxKUhCbB
+YK56ik7HxDk9YyrX3+69dpCFAsOtg3STph4CpKLHmmarJ6rRGJ1gTa5prva7kfZD
+ewYuyjJHhonHPR9WkiVeMm6SV7Odmhw0t8p2TMRx2xre+jdIuHZ7OrPxpd0f+XU9
+NFMzUyccS9WrPbztcWW7c0bTA0wfESCxMQ5HccG3JHy5kVGCj8pugOUk1j1WmTo7
+D+8qlEnRaIaKUfK7Hp+8X8tGtG3D7Dt+N466i9C6zOfCerP0A8hl0c+UQucpy/cg
+G04P7Fpv6b5/yazVReoBcmCFFFPJ04P+MH2h/OIOo4lQjR735H3zRTiCqUpWpQQO
+O6gL8AnoulS7H9IVvUJi1C3HEUnA7GBrkILoi5g94O8NICLq781v7ACUz9gBV6P1
+y5ev/z5Dz9jWbFa4YjY8DQjLPmulYPSJWDmvVEuYHgjvEZTER+fJfJQwMZCQ5bhA
+T4MWh0an1F6H6uTv1Rpc
 -----END X509 CRL-----