-
Notifications
You must be signed in to change notification settings - Fork 727
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4ff67a8
commit bed6ed0
Showing
6 changed files
with
224 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,222 @@ | ||
# Canary Deployments with Helm charts | ||
|
||
This guide shows you how to package a web app into a Helm chart and trigger a canary deployment on upgrade. | ||
|
||
### Packaging | ||
|
||
You'll be be using the [podinfo](https://github.com/stefanprodan/flagger/tree/master/charts/podinfo) chart. | ||
This chart packages a web app made with Go, it's configuration, a horizontal pod autoscaler (HPA) | ||
and the canary configuration file. | ||
|
||
``` | ||
├── Chart.yaml | ||
├── README.md | ||
├── templates | ||
│ ├── NOTES.txt | ||
│ ├── _helpers.tpl | ||
│ ├── canary.yaml | ||
│ ├── configmap.yaml | ||
│ ├── deployment.yaml | ||
│ └── hpa.yaml | ||
└── values.yaml | ||
``` | ||
|
||
### Install | ||
|
||
Create a test namespace with Istio sidecar injection enabled: | ||
|
||
```bash | ||
export REPO=https://github.com/raw/stefanprodan/flagger/master | ||
|
||
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml | ||
``` | ||
|
||
Add Flagger Helm repository: | ||
|
||
```bash | ||
helm repo add flagger https://flagger.app | ||
``` | ||
|
||
Install podinfo with the release name `frontend` (replace `example.com` with your own domain): | ||
|
||
```bash | ||
helm upgrade -i frontend flagger/podinfo \ | ||
--namespace test \ | ||
--set nameOverride=frontend \ | ||
--set backend=http://backend.test:9898/echo \ | ||
--set canary.enabled=true \ | ||
--set canary.istioIngress.enabled=true \ | ||
--set canary.istioIngress.gateway=public-gateway.istio-system.svc.cluster.local \ | ||
--set canary.istioIngress.host=frontend.istio.example.com | ||
``` | ||
|
||
After a couple of seconds Flagger will create the canary objects: | ||
|
||
```bash | ||
# generated by Helm | ||
configmap/frontend | ||
deployment.apps/frontend | ||
horizontalpodautoscaler.autoscaling/frontend | ||
canary.flagger.app/frontend | ||
|
||
# generated by Flagger | ||
configmap/frontend-primary | ||
deployment.apps/frontend-primary | ||
horizontalpodautoscaler.autoscaling/frontend-primary | ||
service/frontend | ||
service/frontend-canary | ||
service/frontend-primary | ||
virtualservice.networking.istio.io/frontend | ||
``` | ||
|
||
After the `frontend-primary` deployment is available, Flagger will scale to zero the `frontend` deployment | ||
and route all traffic to the primary pods. | ||
|
||
Open your browser and navigate to the frontend URL: | ||
|
||
![Frontend](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend.png) | ||
|
||
Now let's install the `backend` release without exposing it outside the mesh: | ||
|
||
```bash | ||
helm upgrade -i backend flagger/podinfo \ | ||
--namespace test \ | ||
--set nameOverride=backend \ | ||
--set canary.enabled=true \ | ||
--set canary.istioIngress.enabled=false | ||
``` | ||
|
||
Check if Flagger has successfully deployed the canaries: | ||
|
||
``` | ||
kubectl -n test get canaries | ||
NAME STATUS WEIGHT LASTTRANSITIONTIME | ||
backend Initialized 0 2019-02-12T18:53:18Z | ||
frontend Initialized 0 2019-02-12T17:50:50Z | ||
``` | ||
|
||
Click on the ping button in the `frontend` UI to trigger a HTTP POST request | ||
that will reach the `backend` app: | ||
|
||
![Jaeger](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend-jaeger.png) | ||
|
||
### Upgrade | ||
|
||
First let's install a load testing service that will generate traffic during analysis: | ||
|
||
```bash | ||
helm upgrade -i flagger-loadtester flagger/loadtester \ | ||
--namepace=test | ||
``` | ||
|
||
Let's enable the load tester and deploy a new `frontend` version: | ||
|
||
```bash | ||
helm upgrade -i frontend flagger/podinfo/ \ | ||
--namespace test \ | ||
--reuse-values \ | ||
--set canary.loadtest.enabled=true \ | ||
--set image.tag=1.4.1 | ||
``` | ||
|
||
Flagger detects that the deployment revision changed and starts the canary analysis along with the load test: | ||
|
||
``` | ||
kubectl -n istio-system logs deployment/flagger -f | jq .msg | ||
New revision detected! Scaling up frontend.test | ||
Halt advancement frontend.test waiting for rollout to finish: 0 of 2 updated replicas are available | ||
Starting canary analysis for frontend.test | ||
Advance frontend.test canary weight 5 | ||
Advance frontend.test canary weight 10 | ||
Advance frontend.test canary weight 15 | ||
Advance frontend.test canary weight 20 | ||
Advance frontend.test canary weight 25 | ||
Advance frontend.test canary weight 30 | ||
Advance frontend.test canary weight 35 | ||
Advance frontend.test canary weight 40 | ||
Advance frontend.test canary weight 45 | ||
Advance frontend.test canary weight 50 | ||
Copying frontend.test template spec to frontend-primary.test | ||
Halt advancement frontend-primary.test waiting for rollout to finish: 1 old replicas are pending termination | ||
Promotion completed! Scaling down frontend.test | ||
``` | ||
|
||
You can monitor the canary deployment with Grafana, open the Flagger dashboard, | ||
select `test` from the namespace dropdown, `frontend-primary` from the primary dropdown and `frontend` from the | ||
canary dropdown. | ||
|
||
![Flagger Grafana Dashboard](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-frontend-dashboard.png) | ||
|
||
Now let's trigger a canary deployment for the `backend` app, but this time you'll change a value in the configmap: | ||
|
||
```bash | ||
helm upgrade -i backend flagger/podinfo/ \ | ||
--namespace test \ | ||
--reuse-values \ | ||
--set canary.loadtest.enabled=true \ | ||
--set httpServer.timeout=25s | ||
``` | ||
|
||
Generate HTTP 500 errors: | ||
|
||
```bash | ||
kubectl -n test exec -it flagger-loadtester-xxx-yyy sh | ||
|
||
watch curl http://backend-canary:9898/status/500 | ||
``` | ||
|
||
Generate latency: | ||
|
||
```bash | ||
kubectl -n test exec -it flagger-loadtester-xxx-yyy sh | ||
|
||
watch curl http://backend-canary:9898/delay/1 | ||
``` | ||
|
||
Flagger detects the config map change and starts a canary analysis. Flagger will pause the advancement | ||
when the HTTP success rate drops under 99% or when the average request duration in the last minute is over 500ms: | ||
|
||
``` | ||
kubectl -n istio-system logs deployment/flagger -f | jq .msg | ||
ConfigMap backend has changed | ||
New revision detected! Scaling up backend.test | ||
Starting canary analysis for backend.test | ||
Advance backend.test canary weight 5 | ||
Advance backend.test canary weight 10 | ||
Advance backend.test canary weight 15 | ||
Advance backend.test canary weight 20 | ||
Advance backend.test canary weight 25 | ||
Advance backend.test canary weight 30 | ||
Advance backend.test canary weight 35 | ||
Halt backend.test advancement success rate 62.50% < 99% | ||
Halt backend.test advancement success rate 88.24% < 99% | ||
Advance backend.test canary weight 40 | ||
Advance backend.test canary weight 45 | ||
Halt backend.test advancement request duration 2.415s > 500ms | ||
Halt backend.test advancement request duration 2.42s > 500ms | ||
Advance backend.test canary weight 50 | ||
Copying backend.test template spec to backend-primary.test | ||
ConfigMap backend-primary synced | ||
Promotion completed! Scaling down backend.test | ||
``` | ||
|
||
![Flagger Grafana Dashboard](https://github.com/raw/stefanprodan/flagger/master/docs/screens/demo-backend-dashboard.png) | ||
|
||
If the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, | ||
the canary is scaled to zero and the rollout is marked as failed. | ||
|
||
```bash | ||
kubectl -n test get canary | ||
|
||
NAME STATUS WEIGHT LASTTRANSITIONTIME | ||
backend Succeeded 0 2019-02-12T19:33:11Z | ||
frontend Failed 0 2019-02-12T19:47:20Z | ||
``` | ||
|
||
If you've enabled the Slack notifications, you'll receive an alert with the reason why the `backend` promotion failed. | ||
|
||
|
||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.