Skip to content

Commit

Permalink
Add tutorial for canaries with Helm
Browse files Browse the repository at this point in the history
  • Loading branch information
stefanprodan committed Feb 12, 2019
1 parent 4ff67a8 commit bed6ed0
Show file tree
Hide file tree
Showing 6 changed files with 224 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/gitbook/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,7 @@
* [Monitoring](usage/monitoring.md)
* [Alerting](usage/alerting.md)

# Tutorials

* [Canary Deployments with Helm](tutorials/canary-helm-chart.md)

222 changes: 222 additions & 0 deletions docs/gitbook/tutorials/canary-helm-chart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
# Canary Deployments with Helm charts

This guide shows you how to package a web app into a Helm chart and trigger a canary deployment on upgrade.

### Packaging

You'll be be using the [podinfo](https://github.com/stefanprodan/flagger/tree/master/charts/podinfo) chart.
This chart packages a web app made with Go, it's configuration, a horizontal pod autoscaler (HPA)
and the canary configuration file.

```
├── Chart.yaml
├── README.md
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── canary.yaml
│   ├── configmap.yaml
│   ├── deployment.yaml
│   └── hpa.yaml
└── values.yaml
```

### Install

Create a test namespace with Istio sidecar injection enabled:

```bash
export REPO=https://github.com/raw/stefanprodan/flagger/master

kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml
```

Add Flagger Helm repository:

```bash
helm repo add flagger https://flagger.app
```

Install podinfo with the release name `frontend` (replace `example.com` with your own domain):

```bash
helm upgrade -i frontend flagger/podinfo \
--namespace test \
--set nameOverride=frontend \
--set backend=http://backend.test:9898/echo \
--set canary.enabled=true \
--set canary.istioIngress.enabled=true \
--set canary.istioIngress.gateway=public-gateway.istio-system.svc.cluster.local \
--set canary.istioIngress.host=frontend.istio.example.com
```

After a couple of seconds Flagger will create the canary objects:

```bash
# generated by Helm
configmap/frontend
deployment.apps/frontend
horizontalpodautoscaler.autoscaling/frontend
canary.flagger.app/frontend

# generated by Flagger
configmap/frontend-primary
deployment.apps/frontend-primary
horizontalpodautoscaler.autoscaling/frontend-primary
service/frontend
service/frontend-canary
service/frontend-primary
virtualservice.networking.istio.io/frontend
```

After the `frontend-primary` deployment is available, Flagger will scale to zero the `frontend` deployment
and route all traffic to the primary pods.

Open your browser and navigate to the frontend URL:

![Frontend](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend.png)

Now let's install the `backend` release without exposing it outside the mesh:

```bash
helm upgrade -i backend flagger/podinfo \
--namespace test \
--set nameOverride=backend \
--set canary.enabled=true \
--set canary.istioIngress.enabled=false
```

Check if Flagger has successfully deployed the canaries:

```
kubectl -n test get canaries
NAME STATUS WEIGHT LASTTRANSITIONTIME
backend Initialized 0 2019-02-12T18:53:18Z
frontend Initialized 0 2019-02-12T17:50:50Z
```

Click on the ping button in the `frontend` UI to trigger a HTTP POST request
that will reach the `backend` app:

![Jaeger](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend-jaeger.png)

### Upgrade

First let's install a load testing service that will generate traffic during analysis:

```bash
helm upgrade -i flagger-loadtester flagger/loadtester \
--namepace=test
```

Let's enable the load tester and deploy a new `frontend` version:

```bash
helm upgrade -i frontend flagger/podinfo/ \
--namespace test \
--reuse-values \
--set canary.loadtest.enabled=true \
--set image.tag=1.4.1
```

Flagger detects that the deployment revision changed and starts the canary analysis along with the load test:

```
kubectl -n istio-system logs deployment/flagger -f | jq .msg
New revision detected! Scaling up frontend.test
Halt advancement frontend.test waiting for rollout to finish: 0 of 2 updated replicas are available
Starting canary analysis for frontend.test
Advance frontend.test canary weight 5
Advance frontend.test canary weight 10
Advance frontend.test canary weight 15
Advance frontend.test canary weight 20
Advance frontend.test canary weight 25
Advance frontend.test canary weight 30
Advance frontend.test canary weight 35
Advance frontend.test canary weight 40
Advance frontend.test canary weight 45
Advance frontend.test canary weight 50
Copying frontend.test template spec to frontend-primary.test
Halt advancement frontend-primary.test waiting for rollout to finish: 1 old replicas are pending termination
Promotion completed! Scaling down frontend.test
```

You can monitor the canary deployment with Grafana, open the Flagger dashboard,
select `test` from the namespace dropdown, `frontend-primary` from the primary dropdown and `frontend` from the
canary dropdown.

![Flagger Grafana Dashboard](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend-dashboard.png)

Now let's trigger a canary deployment for the `backend` app, but this time you'll change a value in the configmap:

```bash
helm upgrade -i backend flagger/podinfo/ \
--namespace test \
--reuse-values \
--set canary.loadtest.enabled=true \
--set httpServer.timeout=25s
```

Generate HTTP 500 errors:

```bash
kubectl -n test exec -it flagger-loadtester-xxx-yyy sh

watch curl http://backend-canary:9898/status/500
```

Generate latency:

```bash
kubectl -n test exec -it flagger-loadtester-xxx-yyy sh

watch curl http://backend-canary:9898/delay/1
```

Flagger detects the config map change and starts a canary analysis. Flagger will pause the advancement
when the HTTP success rate drops under 99% or when the average request duration in the last minute is over 500ms:

```
kubectl -n istio-system logs deployment/flagger -f | jq .msg
ConfigMap backend has changed
New revision detected! Scaling up backend.test
Starting canary analysis for backend.test
Advance backend.test canary weight 5
Advance backend.test canary weight 10
Advance backend.test canary weight 15
Advance backend.test canary weight 20
Advance backend.test canary weight 25
Advance backend.test canary weight 30
Advance backend.test canary weight 35
Halt backend.test advancement success rate 62.50% < 99%
Halt backend.test advancement success rate 88.24% < 99%
Advance backend.test canary weight 40
Advance backend.test canary weight 45
Halt backend.test advancement request duration 2.415s > 500ms
Halt backend.test advancement request duration 2.42s > 500ms
Advance backend.test canary weight 50
Copying backend.test template spec to backend-primary.test
ConfigMap backend-primary synced
Promotion completed! Scaling down backend.test
```

![Flagger Grafana Dashboard](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-backend-dashboard.png)

If the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.

```bash
kubectl -n test get canary

NAME STATUS WEIGHT LASTTRANSITIONTIME
backend Succeeded 0 2019-02-12T19:33:11Z
frontend Failed 0 2019-02-12T19:47:20Z
```

If you've enabled the Slack notifications, you'll receive an alert with the reason why the `backend` promotion failed.



Binary file added docs/screens/demo-backend-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screens/demo-frontend-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screens/demo-frontend-jaeger.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screens/demo-frontend.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit bed6ed0

Please sign in to comment.