Tracing LLM Calls: LangSmith, W&B Weave and Arize

Metrics tell you THAT something is slow or wrong; tracing tells you WHY, by capturing the full record of what happened inside each request. This matters far more for LLM apps than for ordinary services because a single user request often fans out into many steps — a RAG pipeline does an embedding, a vector search, a rerank, then a generation; an agent loops through several tool calls and LLM turns. When the final answer is bad, you need to see every intermediate step to find where it went wrong.

A trace is a tree of spans. The top-level span is the whole request; nested child spans are the individual steps (each LLM call, retrieval, tool invocation). For each span you capture the exact inputs and outputs (the actual prompt sent and completion received — not a summary), metadata (model, temperature, token counts), latency, cost, and any errors. Crucially, tracing records the RESOLVED prompt after all templating and context injection — so when an answer is wrong, you can see the literal text the model actually received, which is usually where bugs hide (a mis-formatted template, empty retrieved context, a truncated history).

Three tools dominate. LangSmith (from the LangChain team) traces chains and agents step by step, integrates tightly with LangChain but works with any stack, and pairs tracing with datasets and evaluation. Weights & Biases Weave brings W&B's experiment-tracking heritage to LLM apps: you decorate functions and it logs their inputs/outputs into a queryable trace tree, strong when you also want to track evaluations and compare versions. Arize Phoenix is an open-source, OpenTelemetry-based tracing and evaluation tool with a focus on production monitoring and built on open standards, so it slots into existing observability pipelines.

A practical note on standards: OpenTelemetry (via the OpenLLMetry / OpenInference conventions) is emerging as a vendor-neutral way to emit LLM traces, so you can instrument once and send traces to whichever backend you prefer. And be deliberate about capturing full inputs/outputs: they're invaluable for debugging and building eval datasets, but they contain user data — so you sample, redact PII, and set retention policies. The payoff is that when a user reports a bad answer, you pull up its exact trace and see the whole causal chain instead of guessing.

# A minimal decorator-based tracer that records a span tree with the
# inputs, outputs, latency, and token/cost metadata these tools capture.
import time
import functools
import contextvars

_current_parent = contextvars.ContextVar("parent", default=None)
TRACE = []  # flat list of spans; parent_id links them into a tree

def traced(name: str):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            span = {"name": name, "parent": _current_parent.get(),
                    "input": {"args": args, "kwargs": kwargs}}
            token = _current_parent.set(id(span))
            start = time.perf_counter()
            try:
                out = fn(*args, **kwargs)
                span["output"] = out
                return out
            finally:
                span["latency_ms"] = (time.perf_counter() - start) * 1000
                TRACE.append(span)
                _current_parent.reset(token)
        return wrapper
    return deco

@traced("retrieve")
def retrieve(q): return ["doc about " + q]

@traced("generate")
def generate(q, ctx): return f"Answer to {q!r} using {ctx}"

@traced("rag_pipeline")            # parent span wrapping the child steps
def rag(q):
    ctx = retrieve(q)
    return generate(q, ctx)

rag("vector databases")
for s in TRACE:
    print(f"{s['name']:<12} {s['latency_ms']:.2f}ms parent={s['parent']}")