Complete PromQL queries for a per-second 5xx error rate and 95th-percentile latency.
You have a Prometheus counter http_requests_total labelled by status, and a histogram http_request_duration_seconds whose _bucket series carry the le label.
Complete two PromQL (Prometheus query language) expressions:
# 1) per-second 5xx error rate over the last 5 minutes
sum(rate(______________________[5m]))
# 2) 95th-percentile request latency over 5 minutes
histogram_quantile(0.95, ______________________)Fill in both blanks. Do not average latency, and do not average percentiles.
Error rate is rate() over the http_requests_total counter filtered to 5xx and summed over 5m. For p95 use histogram_quantile(0.95, ...) over the rate of the histogram _bucket series, but you must sum by (le) those buckets first. Never average latency, and never average percentiles across instances — aggregate buckets, then take the quantile.
- ✗Averaging latency, or averaging each instance's percentile, instead of aggregating buckets
- ✗Feeding histogram_quantile the counter total rather than the rate of _bucket series
- ✗Forgetting
sum by (le)so buckets from many instances are not combined
- →Why must you
sum by (le)beforehistogram_quantilewhen scraping many instances? - →Why is averaging two instances' p95 values not the fleet-wide p95?
Solution
Two metrics, two techniques. Errors are a rate() over the counter filtered by status. Latency is a quantile over the histogram buckets, never an average.
# 1) per-second 5xx error rate over 5m — rate() the counter, keep 5xx, sum:
sum(rate(http_requests_total{status=~"5.."}[5m]))
# 2) p95 latency over 5m — sum buckets by le, THEN take the quantile:
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))Why sum by (le). Every instance exposes its own _bucket series. histogram_quantile reconstructs the percentile from the bucket shape, so the buckets are summed by the le label first — otherwise the quantile is per-instance. Why not an average. A mean latency hides the tail: a 20ms mean can hide a 2s p99. And percentiles cannot be averaged across instances — the fleet p95 is derived from the combined buckets, not from an average of each p95.
