What to Monitor: Latency, Throughput, Tokens, Errors and Cost
~13 min read
LLM observability starts with the operational metrics unique to generation: TTFT and total latency, tokens-per-second throughput, token usage, error/refusal rates, and cost per request.
What to Monitor: Latency, Throughput, Tokens, Errors and Cost is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Split latency into time-to-first-token (perceived responsiveness), total time, and inter-token latency — and track p95/p99, not just averages
- •Track throughput as tokens-per-second (TPS) across the fleet — the honest capacity measure since request lengths vary widely
- •Monitor input/output token usage: it drives cost and flags behavior changes (rambling outputs, bloating context)
- •Go beyond HTTP errors: track timeouts, rate limits, unparseable outputs, and refusals — failures invisible to generic monitoring
- •Attribute cost per request/feature/customer and alert on drift up — often the earliest sign of prompt bloat or a bad routing change