Add support for custom metrics

[stefanprodan]

This PR adds a query field to the Canary CRD and implements custom metric checks. Flagger will run the promql query, converts the result to float64 and compares it with the metric threshold. This allows for arbitrary Prometheus scalar queries to be used in the analysis process.

Example:

  canaryAnalysis:
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: "404s percentage"
      threshold: 5
      interval: 1m
      query: |
        100 - sum(
            rate(
                istio_requests_total{
                  reporter="destination",
                  destination_workload_namespace="test",
                  destination_workload="podinfo",
                  response_code!="404"
                }[1m]
            )
        )
        /
        sum(
            rate(
                istio_requests_total{
                  reporter="destination",
                  destination_workload_namespace="test",
                  destination_workload="podinfo"
                }[1m]
            )
        ) * 100

The above config validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent. If the 404s rate reaches the 5% threshold the canary fails.

Failed check event:

Halt podinfo.test advancement 404s percentage 52.84 > 5

Fix: #59
Fix: #8

[codecov-io]

Codecov Report

Merging #60 into master will decrease coverage by 1.43%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage   56.88%   55.45%   -1.44%     
==========================================
  Files           9        9              
  Lines        1350     1385      +35     
==========================================
  Hits          768      768              
- Misses        448      481      +33     
- Partials      134      136       +2

Impacted Files	Coverage Δ
pkg/controller/observer.go	53.33% <0%> (-10.67%)	⬇️
pkg/controller/scheduler.go	39.33% <0%> (-2.3%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04a56a3...2e351fc. Read the comment docs.