Observability, Evaluation, and User Feedback Systems

Observability: Every Request Emits a Trace

A structured trace captures the full context of each request for debugging, evaluation, and improvement:

trace = {
    'request_id': str(uuid.uuid4()),
    'user_id': user_id,
    'feature': 'product_assistant',
    'query': user_query,
    'model': 'claude-sonnet-4-6',
    'tier': 2,
    'cache_hit': False,
    'context': {
        'docs_retrieved': 10,
        'docs_reranked': 3,
        'memory_items': 2,
    },
    'latency_ms': {
        'guardrail_in': 12,
        'cache_lookup': 8,
        'retrieval': 45,
        'reranking': 120,
        'model_call': 1840,
        'guardrail_out': 15,
        'total': 2040,
    },
    'tokens': {'input': 2100, 'output': 380},
    'cost_usd': 0.0042,
    'response': response_text,
}

Traces flow to an observability backend (LangSmith, Arize, custom) where you:

• Monitor latency percentiles (P50, P95, P99) per component
• Track cost per feature, per user, per day
• Sample traces for manual review and eval

Async Evaluation Pipeline

Run evaluations asynchronously — don't block the response path:

1. Request completes → trace written to queue
2. Evaluator worker picks up trace
3. Runs LLM-as-judge or rule-based eval
4. Writes eval score back to trace store
5. Dashboard aggregates scores per feature/day

User Feedback Systems (Chip Huyen, Ch.10)

Feedback types, ranked by quality:

1. Expert annotation (highest quality, lowest volume): humans evaluate a sample of traces on a rubric. Expensive but ground truth.

2. Explicit user feedback (medium quality, low volume): thumbs up/down, star ratings, 'flag this response'. Biased toward extremes (very good or very bad). Positive skew from politeness; negative skew from frustration.

3. Implicit behavioral signals (lower quality, high volume):

   • Copy/paste: user copied the response → likely positive
   • Re-ask or rephrasing immediately after: → likely negative
   • Click-through on a recommendation: → positive
   • Session abandonment after response: → possibly negative
   • 'That's wrong' / 'Actually...' as next message: → negative

4. Conversational feedback extraction: parse the next user message for quality signals using a classifier: 'Perfect, thanks!' → positive 'That's not what I meant' → negative 'Can you try again?' → negative 'Interesting, tell me more' → positive

Limitations of feedback systems:

• Selection bias: users who leave feedback are not representative
• Copy-paste ambiguity: copied to edit (neutral) vs. copied to use (positive)
• Feedback gaming: users learn the UI and game ratings
• Response bias: users rate 'nice but wrong' responses positively

Closing the Loop: Traces + feedback → curated evaluation dataset → prompt engineering or fine-tuning → deploy improved system → re-measure on held-out eval set. Repeat weekly.