LLM Observability: Metrics & Telemetry
Learn observability requirements, key performance metrics (TPS, TTFT), and integrate telemetry tools.
Establish Transport pipe connection
Host client (IDE or AI assistant) connects to the server process over standard output streams.
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Contrast evaluation (pre-deployment) and observability (post-deployment)
- •Monitor performance metrics: Token-per-second, Time-to-first-token, cost, errors
- •Integrate observability logging using telemetry tracing tools
What is it?
LLM observability is the instrumentation layer that turns a production LLM system from a black box into something you can actually debug — tracking latency (especially time-to-first-token), token throughput, cost per request, and error/refusal rates, then tracing each request's full lifecycle across retrieval, tool calls, and generation so a bad answer can be traced back to exactly where it went wrong. It also covers detecting silent quality drift over time and alerting the right people before users notice a problem.
Why it exists
Agent systems execute complex non-deterministic loops. Without detailed nested tracing, it is impossible to diagnose why an agent failed or where latency was wasted.
Problem it solves
Diagnosing agent failures, latency tracing, token volume cost monitoring, and capturing user feedback.
Intuition
Once the AI is live and being used by real people, we must monitor it to make sure it doesn't break, run slowly, or cost too much. This is called "Observability". We track how fast it answers (Time to First Token), how many tokens it uses, and if users are clicking "thumbs up" or "thumbs down" on its answers.
Analogy
Think of observability like a flight data recorder (black box) on an airplane. If the plane crashes (agent gets stuck in a loop), you do not just check if the engine was on; you review the exact timeline of cockpit decisions, sensor readings, and steering adjustments.
Technical explanation
Observability architectures trace agent trajectories and tool execution spans. We implement OpenTelemetry standards to export tracing data. OpenTelemetry captures spans for LLM calls, vector database lookups, and code execution blocks. This allows tracing nested execution graphs, locating where latency spikes or token leakages occur in long-running agent loops.
Architecture
Telemetry tracing nodes sending spans (traces) asynchronously over network proxies to collector storage databases (Opik/Phoenix).
Workflow
- Decorate agent step -> 2. Open span context -> 3. Log input payload -> 4. Execute step -> 5. Export span -> 6. Compute metrics.
Example
import time
trace decorator example
def trace_step(func): def wrapper(*args): start = time.time() res = func(*args) print(f"Span {func.name} took {time.time()-start:.4f}s") return res return wrapper
Real-world usage
Integrating Arize Phoenix dashboards to trace and debug multi-agent code assistants.
Trade-offs
Full span logging is diagnostic but expensive; sample tracing is cost-effective but misses outlier bugs.
Visual explanation
Nested Agent Telemetry Spans: ┌────────────────────────────────────────────────────────┐ │ Span: Complete User Request (Duration: 3.2s) │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Span: Vector Search (Duration: 0.2s) │ │ │ └──────────────────────────────────────────────────┘ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Span: LLM Agent Turn 1 (Duration: 1.5s) │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ Span: Tool execution │ (Duration: 0.6s) │ │ │ │ └────────────────────────┘ │ │ │ └──────────────────────────────────────────────────┘ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Span: LLM Agent Turn 2 (Duration: 1.5s) │ │ │ └──────────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────────┘
Advantages
- —
Exposes exact failure steps in multi-turn runs
- —
Provides real-time dashboards of costs and latency metrics
Disadvantages
- —
Logging every intermediate token adds network bandwidth overhead
Common mistakes
- —
Redacting output text but forgetting to remove sensitive API keys or passwords from metadata spans
- —
Not logging intermediate thought blocks (makes troubleshooting reasoning loops impossible)
🎤 Interview questions
How does OpenTelemetry collect and export spans? Why is asynchronous telemetry batching important in production?
How would you detect and alert on drift in model output quality or factual accuracy using live traffic monitoring?
📂 Subtopics
What to Monitor: Latency, Throughput, Tokens, Errors and Cost
LLM observability starts with the operational metrics unique to generation: TTFT and total latency, tokens-per-second throughput, token usage, error/refusal rates, and cost per request.
~13 min
Tracing LLM Calls: LangSmith, W&B Weave and Arize
Tracing captures the full story of each request — inputs, outputs, prompts, tool calls, latency and cost — across every step of a chain or agent. LangSmith, W&B Weave and Arize Phoenix are the common tools.
~13 min
Detecting Drift: Quality Degradation and Prompt Sensitivity
LLM apps degrade silently — inputs shift, a model version changes under you, a prompt edit backfires. Drift detection watches output quality and input distributions over time to catch slow decay before users do.
~12 min
Alerting and Dashboards: Thresholds and What to Watch
Observability is only useful if someone acts on it. Good dashboards surface the key metrics at a glance, and good alerts fire on the few conditions that genuinely need a human — without drowning the team in noise.
~12 min