Alerting and Dashboards: Thresholds and What to Watch

Collecting metrics, traces, and drift signals is wasted effort if nobody sees the problem in time. The last mile of observability is presentation and alerting: dashboards for humans to scan, and alerts for the machine to page someone when something crosses a line. Both are easy to do badly.

A good LLM dashboard is layered. A top-level 'health' view shows the vital signs at a glance: request rate, p95 TTFT and total latency, error/refusal rate, tokens per second, and current cost per hour. Below that, breakdowns let you localize a problem — by model, by feature/endpoint, by customer — so when the top-level error rate spikes you can immediately see it's isolated to one model or one tenant. A quality panel tracks the eval-score and drift trends over time. And a link from any anomaly down to example traces closes the loop from 'something's wrong' to 'here are the exact requests that were wrong.' The discipline is to put the few numbers that matter up front, not to build a wall of 50 charts nobody reads.

Alerting is where teams most often go wrong, in two opposite directions. Alert on too much and you get alert fatigue — people start ignoring pages, and the real incident gets missed in the noise. Alert on too little and you find out about outages from users. The craft is choosing the few conditions that genuinely require human action, and setting thresholds so they fire on real problems, not normal variance.

Practical guidelines for LLM alerts: alert on symptoms users feel (p95 latency above target, error/refusal rate above baseline, TTFT regression) rather than on every raw metric. Prefer rate-of-change and sustained breaches over instantaneous spikes — 'error rate above 5% for 5 minutes' is far more actionable than a single momentary blip. Set thresholds off your healthy baseline and historical percentiles, not arbitrary round numbers. Include cost alerts (a sudden jump in cost per request or hourly spend often signals a bad deploy — prompt bloat, a routing bug, or a runaway loop). Attach context to every alert: which service, which model, a link to the dashboard and to example traces, so the on-call engineer starts debugging instead of hunting. And define severity — page a human for user-facing outages, send quality-drift warnings to a channel for next-business-day review.

The goal is a system where a healthy day is quiet, a real problem produces exactly one clear, actionable alert with a path straight to the offending traces, and the dashboard answers 'is everything okay?' in five seconds.

# A sustained-breach alerting rule (avoids paging on momentary spikes)
# plus a small dashboard-summary snapshot of the vital signs.
from collections import deque
import time

class SustainedThresholdAlert:
    """Fires only if the metric stays over the threshold for `for_seconds`,
    so a single blip doesn't page anyone."""
    def __init__(self, name, threshold, for_seconds=300):
        self.name, self.threshold, self.for_seconds = name, threshold, for_seconds
        self.breaching_since = None

    def update(self, value, now=None) -> str | None:
        now = now or time.time()
        if value > self.threshold:
            self.breaching_since = self.breaching_since or now
            if now - self.breaching_since >= self.for_seconds:
                return (f"ALERT[{self.name}]: {value:.3f} > {self.threshold} "
                        f"sustained {self.for_seconds}s -> page on-call")
        else:
            self.breaching_since = None   # recovered, reset
        return None

err_alert = SustainedThresholdAlert("error_rate", threshold=0.05, for_seconds=300)
print(err_alert.update(0.09, now=0))      # breach starts, no page yet
print(err_alert.update(0.09, now=301))    # sustained 5 min -> page

def dashboard_snapshot(metrics: dict) -> str:
    """The five-second 'is everything okay?' view."""
    return (f"req/s={metrics['rps']}  p95_ttft={metrics['p95_ttft_ms']}ms  "
            f"err={metrics['error_rate']:.1%}  tps={metrics['tps']}  "
            f"$/hr={metrics['cost_per_hr']:.2f}")

print(dashboard_snapshot({"rps": 42, "p95_ttft_ms": 210,
    "error_rate": 0.012, "tps": 1850, "cost_per_hr": 3.10}))