synapse_http_server_response_time_seconds_bucket has too high cardinality

[matrixbot]

This issue has been migrated from #11082.

---

Description

This issue was discovered while looking at other problematic time series due to #11081. There's a similar cardinality issue here, caused by the multiplying factor between the code, servlet and tag labels on the series.

Also, it's problematic to have time series that are part of the same bucket but don't all have the same set of labels. There's only a few series that have the tag label. That should probably be a separate histogram as the label difference will cause aggregation issues.

Steps to reproduce

• Run lots of Synapse instances
• Scrape them all with a single Prometheus instance
• Prometheus gets sad

Version information

• Homeserver: any
• Version: any since this metric got introduced
• Install method: unrelated
• Platform: unrelated