What to Monitor: Latency, Throughput, Tokens, Errors and Cost

You can't improve or debug what you don't measure, and LLM systems need metrics that ordinary web-service dashboards don't capture. The first job of observability is deciding WHAT to track. The essential set breaks into five families.

Latency, split two ways. Because generation is incremental, a single 'response time' number hides what matters. Track time to first token (TTFT) — how long until the user sees anything, the metric that governs perceived responsiveness — separately from total generation time and from inter-token latency (the pace of streaming). And always track distributions, not just averages: p50, p95, p99. The p99 tail is where users actually feel pain, and averages hide it.

Throughput measures capacity: requests per second, and more tellingly for LLMs, tokens per second (TPS) across the fleet. TPS is the honest measure of how much work your GPUs are doing, since requests vary enormously in length. Watching TPS against your theoretical GPU capacity tells you how much headroom remains.

Token usage is both a cost driver and a behavior signal. Track input tokens, output tokens, and their ratio per request and in aggregate. A sudden rise in average output tokens might mean the model started rambling after a prompt change; growing input tokens might mean context/history is bloating. Tokens are the unit that connects performance, cost, and quality.

Error and quality-failure rates go beyond HTTP 500s. For LLMs you also monitor timeouts, rate-limit rejections, malformed/unparseable outputs (e.g. broken JSON when you asked for JSON), refusal rates, and safety-filter triggers. A spike in unparseable outputs or refusals is a real incident even though every request returned HTTP 200 — a class of failure invisible to generic monitoring.

Cost per request ties it to the business. Since you're tracking tokens and model choice, you can attribute spend — per request, per feature, per customer — and alert when cost per request drifts up (often the earliest sign of prompt bloat or a bad routing change). Together these five families give you the operational picture; the next subtopics cover HOW you capture them (tracing) and what to do when they move (drift, alerting).

# A lightweight per-request metrics record capturing the five families,
# plus percentile computation (the numbers that actually matter).
import time
from dataclasses import dataclass, asdict

@dataclass
class RequestMetrics:
    ttft_ms: float          # time to first token (perceived latency)
    total_ms: float         # full generation time
    input_tokens: int
    output_tokens: int
    model: str
    error: str | None = None   # None, 'timeout', 'unparseable', 'refusal', ...

    @property
    def tokens_per_sec(self) -> float:
        return self.output_tokens / (self.total_ms / 1000) if self.total_ms else 0.0

    @property
    def cost_usd(self) -> float:
        price = {"gpt-4o": (2.5e-6, 1e-5), "gpt-4o-mini": (1.5e-7, 6e-7)}
        pin, pout = price.get(self.model, (0, 0))
        return self.input_tokens * pin + self.output_tokens * pout

def percentile(values: list[float], p: float) -> float:
    if not values:
        return 0.0
    s = sorted(values)
    return s[min(len(s) - 1, int(len(s) * p / 100))]

m = RequestMetrics(ttft_ms=180, total_ms=4200, input_tokens=850,
                   output_tokens=320, model="gpt-4o-mini")
print(asdict(m))
print(f"TPS={m.tokens_per_sec:.1f}  cost=${m.cost_usd:.5f}")
print("p95 TTFT:", percentile([120, 180, 90, 400, 210, 950], 95), "ms")