Create a set of template Multiwindow, Multi-Burn-Rate Alerts, tailored to an API view

[david-martin]

Multiwindow, Multi-Burn-Rate as per https://sre.google/workbook/alerting-on-slos/

Suggested SLIs

• Availability based on error rate (5xx responses) from an API as a whole
• Availability based on latency from an API as a whole

Both of these will include example prometheus recording rules for how to set up these alerts.

We can use https://sloth.dev/ to help generate the rules.

We should include prometheus unit tests for these and include manual testing if possible.

It will more than likely require istio or envoy metrics (and so be specific to that gateway api provider),
however that's reasonable as there are no standards around gateway api low level metrics.

It is important to have these be templated as we have no way of knowing the exact criteria for alerting that will suit every individual user.
We can use what we think are reasonable default values, and allow them to be templated where possible.