AI System Architecture Patterns
Capstone view of production AI systems: how context construction, guardrails, model routers/gateways, semantic caching, agent orchestration, observability, and user feedback loops fit together into one coherent, operable architecture. Grounded in Chip Huyen's AI Engineering Ch.10 framework.
GPU Queue Strategy:
Client request queue initialization
Requests from multiple users enter the scheduler queue.
▶📚 Prerequisites(5)
🎓 Learning objectives
- •Draw the end-to-end request path through a production AI system: ingress → guardrail → cache → model gateway → response pipeline → observability
- •Explain how context construction (retrieval, memory, tool results) feeds into the model call and what determines context quality
- •Design a model router/gateway that selects between models based on task complexity, cost, and latency targets
- •Describe semantic caching and when it reduces latency without degrading quality
- •Identify where agent orchestration adds concurrency and how to bound its blast radius in production
- •Design an observability stack covering traces, metrics, and evaluation signals for a multi-step AI system
- •Explain how to extract user feedback from conversation signals and use it to close the improvement loop
What is it?
A production AI system is not a single model call — it is a pipeline of components that together transform a user request into a reliable, safe, and efficient response. Chip Huyen (AI Engineering, Ch.10) defines the architecture around five concerns:
- Context construction — assembling the right information (retrieval, memory, tool results, conversation history) before the model call
- Safety and guardrails — filtering inputs and outputs before they reach users
- Model routing and gateways — directing requests to the right model at the right cost
- Latency reduction — caching, streaming, and concurrency to meet SLA targets
- Observability and feedback — tracing, evaluating, and improving the system over time
These concerns do not sit in separate services — they are layered checkpoints in a single request path. Understanding their interaction is the capstone skill for AI engineers.
Why it exists
A bare LLM API call works in demos. It breaks in production for five reasons:
Reliability: LLMs hallucinate, go off-topic, or produce unsafe outputs without guardrails. A production system must catch these failures before they reach users.
Cost: Without caching and routing, every request hits the most expensive model. Real systems use a cascade of cheaper models for simple queries.
Latency: A 3-second TTFB is unacceptable for real-time interfaces. Semantic caches, streaming responses, and pre-fetching shave seconds off the critical path.
Quality: Without retrieval and context engineering, the model answers from stale training data. The right context changes everything.
Improvement: A system you can't observe is a system you can't improve. Traces, evals, and user feedback loops are what let you close the quality gap over time.
Chip Huyen (Ch.10): 'The model is the least interesting part of most production AI systems. The architecture around it determines whether it succeeds.'
Problem it solves
- Users are getting hallucinated answers — the system has no factual grounding and no output guardrails.
- The AI feature is too expensive to scale — every request hits GPT-4 even for simple queries.
- P95 latency is 8 seconds — there's no caching, no streaming, and context assembly is blocking.
- After launch, quality is declining — there are no traces, no evals, no feedback loop.
- The agent is causing unintended side effects — no blast-radius controls on tool calls.
- Users are not reporting problems — there's no mechanism to capture implicit feedback signals.
Intuition
Think of a production AI system as an assembly line with quality checkpoints.
A user request enters the line and passes through stations:
Station 1 — Input Guardrail: Is this request safe? On-topic? Does it violate policy? Reject or transform it before spending compute.
Station 2 — Semantic Cache: Have we answered this (or something very similar) recently? Return the cached result. Skip the rest of the line.
Station 3 — Context Construction: Retrieve documents, look up memory, fetch tool results. Assemble the fullest possible context for the model.
Station 4 — Model Router: Which model fits this request? A cheap fast model for summarization, an expensive reasoning model for complex analysis.
Station 5 — Model Call (+ Agent Loop): The actual inference. If agents are involved, this station loops — calling tools, receiving results, reasoning again.
Station 6 — Output Guardrail: Does the response contain PII, policy violations, or hallucinated citations? Transform or block before delivery.
Station 7 — Observability: Log the trace. Evaluate the output. Record implicit feedback.
Each station is optional — not every system needs all of them — but together they turn a fragile demo into a reliable, improvable product.
Analogy
A production AI system is like an air traffic control tower for model calls.
The control tower (gateway) doesn't fly the planes (models) — it decides which runway they land on, holds them if it's unsafe, reroutes them if a runway is congested, and logs everything for the black box.
The pilots (models) are capable, but they don't decide the routing. The tower has the full picture: available runways (models), weather conditions (latency/cost), and safety rules (guardrails). Without the tower, every pilot makes independent decisions and the system becomes chaotic. With it, the system is orchestrated, auditable, and recoverable when something goes wrong.
Technical explanation
CONTEXT CONSTRUCTION (Chip Huyen, Ch.10):
Context quality determines answer quality more than model capability for knowledge-intensive tasks. The context pipeline has three layers:
- Retrieval: vector search over knowledge bases, filtered by metadata (recency, source, topic)
- Reranking: cross-encoder reranker selects top-K documents from retrieved candidates
- Memory: user profile (preferences, history) + session memory (prior turns)
Context budget management: LLMs have finite context windows. The system must prioritize: Priority 1: System prompt / task instructions Priority 2: Retrieved documents (most relevant first) Priority 3: Conversation history (summarize if too long) Priority 4: Tool results
GUARDRAILS:
Input guardrails run BEFORE the model call (cheap to run, saves expensive compute):
- PII detection: regex + NER to detect/redact names, SSNs, card numbers
- Prompt injection: classifier trained to detect 'ignore previous instructions' patterns
- Topic classifier: is this in scope for this AI assistant?
- Rate limiter: per-user, per-IP, per-key limits
Output guardrails run AFTER the model call:
- Citation grounding: are factual claims supported by retrieved documents?
- Safety classifier: toxic content, self-harm references, illegal advice
- Format validator: does the JSON conform to expected schema?
- PII re-check: ensure no PII leaked from context into response
MODEL ROUTER / GATEWAY:
A router classifies each incoming request and dispatches to the appropriate tier: Simple → Tier 1 (fast, cheap, e.g., Haiku / GPT-4o-mini): ~$0.001/1K tokens Moderate → Tier 2 (capable, e.g., Sonnet / GPT-4o): ~$0.003-0.015/1K tokens Complex → Tier 3 (powerful, e.g., Opus / o3): ~$0.015-0.075/1K tokens
Router approaches: a. Rule-based: if query length < 50 tokens AND no code → Tier 1 b. Classifier-based: fine-tuned classifier on (query, task_type) → tier label c. Cascade: try Tier 1 first; if confidence < threshold, retry with Tier 2
The gateway also handles: retries with exponential backoff, fallback to alternate provider, load balancing across API keys, request deduplication, and cost tracking per feature/user.
SEMANTIC CACHING:
Semantic cache stores (query_embedding, response) pairs. At request time:
- Embed the incoming query
- Search the cache for nearest neighbor query embeddings
- If max cosine similarity > threshold (e.g., 0.92): return cached response
- Otherwise: proceed to model call, store result in cache
When to use: high-repetition query patterns (product FAQs, support bots, search assistants) When to avoid: queries that require fresh data (news, prices, personalized responses) Cache invalidation: TTL-based (FAQ cache: 24h), event-based (clear on knowledge base update)
AGENT ORCHESTRATION:
Agents add concurrency (run independent tools in parallel) but also risk:
- Blast radius: a misconfigured tool call can delete data, send emails, charge cards
- Cost runaway: uncapped loops hit rate limits and burn budget
- Latency variance: tool call chains are hard to predict
Production controls:
- Max iterations: hard cap (e.g., 20 steps)
- Time budget: abort after N seconds regardless of progress
- Cost budget: per-session token and API call limit
- Permission tiers: read-only tools vs. write tools vs. irreversible tools
- Human-in-the-loop: interrupt before high-risk actions (send email, delete record)
OBSERVABILITY AND FEEDBACK:
Every request should emit a structured trace:
- Request ID (correlation across components)
- Input tokens, output tokens, model used, tier, cost
- Latency breakdown: cache lookup, retrieval, model call, guardrail
- Retrieval metrics: documents retrieved, reranked, included
- Output quality: async LLM-as-judge score, citation precision
User feedback systems (Chip Huyen, Ch.10): Explicit: thumbs up/down, star ratings, flagging — low volume, high signal Implicit: copy-paste (positive), re-ask / rephrasing (negative), click-through (positive) Conversational: extracting quality signals from the next message in a conversation ('That's wrong, try again' → failure signal; 'Perfect, thanks' → success signal)
Limitations of feedback systems:
- Selection bias: users who give feedback are not representative
- Negative feedback skew: people are more likely to report bad than good
- Implicit signal noise: copy-paste could mean the user is going to edit it — not pure praise
- Gaming: users learn that thumbs-up gets better responses, inflating positive signals
Feedback loop: traces + feedback → evaluation dataset → fine-tuning or prompt updates → improved model
Architecture
Three-Tier AI Production Stack:
┌─────────────────────────────────────────────────────────────┐ │ TIER 1: EDGE / GATEWAY LAYER │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ Rate Limiter │ │ Auth/AuthZ │ │ Input Guardrail │ │ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Semantic Cache (Redis + vector index) │ │ │ └──────────────────────────────────────────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ cache miss ┌────────────────────────────▼────────────────────────────────┐ │ TIER 2: ORCHESTRATION LAYER │ │ │ │ ┌──────────────────────┐ ┌──────────────────────────────┐ │ │ │ Context Constructor │ │ Model Router / Gateway │ │ │ │ • RAG retrieval │ │ • Complexity classifier │ │ │ │ • Memory lookup │ │ • Tier 1/2/3 dispatch │ │ │ │ • Tool pre-fetch │ │ • Retry / fallback │ │ │ └──────────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Agent Orchestrator (if agentic) │ │ │ │ • Tool dispatch (concurrent where independent) │ │ │ │ • Budget enforcement (iterations, time, cost) │ │ │ │ • Human-in-the-loop checkpoints │ │ │ └──────────────────────────────────────────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ ┌────────────────────────────▼────────────────────────────────┐ │ TIER 3: OUTPUT + OBSERVABILITY LAYER │ │ │ │ ┌──────────────────────┐ ┌──────────────────────────────┐ │ │ │ Output Guardrail │ │ Observability Pipeline │ │ │ │ • Citation check │ │ • Structured trace emit │ │ │ │ • Safety filter │ │ • Async LLM eval │ │ │ │ • Format validation │ │ • Feedback capture │ │ │ └──────────────────────┘ └──────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘
Workflow
Designing a production AI system (Chip Huyen's framework):
-
Define the request path on paper:
- What guardrails does this use case need? (Safety? PII? Topic?)
- What context does the model need? (RAG? Memory? Tools?)
- Is this agentic or single-turn?
-
Build the context pipeline first — it has the highest quality leverage. a. Identify knowledge sources (docs, databases, APIs) b. Choose retrieval strategy (vector search, keyword, hybrid) c. Add reranker if retrieval precision is insufficient d. Define context budget and priority order
-
Add guardrails incrementally: a. Start with a blocklist + basic topic classifier (input) b. Add citation grounding check (output) for factual assistants c. Add PII detection if handling sensitive data
-
Build the model gateway: a. Start with a single model — no router needed b. Add a cheaper Tier 1 model for the subset of simple queries c. Implement retry with exponential backoff and provider fallback
-
Add semantic caching after you understand query distribution: a. Log 1,000 production queries b. Cluster them to find high-repetition patterns c. Set similarity threshold via human evaluation of 50 cache-hit pairs
-
Instrument observability from day one: a. Emit structured trace on every request (latency, cost, model, tokens) b. Add async LLM-as-judge eval on a 1% sample c. Integrate feedback capture (thumbs, implicit signals)
-
Close the feedback loop: a. Review low-rated sessions weekly b. Add failure cases to evaluation dataset c. Update prompts / retrieval / guardrails based on patterns
Example
# Simplified production AI request handler import time import uuid from anthropic import Anthropic client = Anthropic() # --- Minimal semantic cache (in-memory, replace with Redis + FAISS in prod) --- import numpy as np from typing import Optional CACHE: list[tuple[np.ndarray, str]] = [] # (embedding, response) CACHE_THRESHOLD = 0.92 def cosine_sim(a: np.ndarray, b: np.ndarray) -> float: return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) def embed_query(text: str) -> np.ndarray: # In production: call your embedding provider # Here: stub returning a random unit vector v = np.random.randn(1536) return v / np.linalg.norm(v) def cache_lookup(q_emb: np.ndarray) -> Optional[str]: for emb, resp in CACHE: if cosine_sim(q_emb, emb) >= CACHE_THRESHOLD: return resp return None # --- Minimal input guardrail --- BLOCKLIST = ['ignore previous instructions', 'you are now', 'jailbreak'] def input_guardrail(query: str) -> tuple[bool, str]: q_lower = query.lower() for phrase in BLOCKLIST: if phrase in q_lower: return False, 'Request blocked by safety policy.' return True, '' # --- Minimal model router --- def select_model(query: str) -> str: # Simple heuristic: short queries without code → fast model if len(query) < 150 and '```' not in query: return 'claude-haiku-4-5-20251001' # Tier 1: fast, cheap return 'claude-sonnet-4-6' # Tier 2: capable # --- Main request handler --- def handle_request(user_query: str, context_docs: list[str] = []) -> dict: trace = { 'request_id': str(uuid.uuid4()), 'query': user_query, 'timestamp': time.time(), } # 1. Input guardrail safe, reason = input_guardrail(user_query) if not safe: trace['blocked'] = True trace['block_reason'] = reason return {'response': reason, 'trace': trace} # 2. Semantic cache t0 = time.time() q_emb = embed_query(user_query) cached = cache_lookup(q_emb) if cached: trace['cache_hit'] = True trace['latency_ms'] = int((time.time() - t0) * 1000) return {'response': cached, 'trace': trace} trace['cache_hit'] = False # 3. Context construction (simplified: use passed docs) context = '\n\n'.join(context_docs[:3]) # top 3 docs system_prompt = ( 'You are a helpful assistant. ' 'Answer based on the provided context. ' f'Context:\n{context}' ) if context else 'You are a helpful assistant.' # 4. Model routing model = select_model(user_query) trace['model'] = model # 5. Model call t1 = time.time() response = client.messages.create( model=model, max_tokens=1024, system=system_prompt, messages=[{'role': 'user', 'content': user_query}], ) answer = response.content[0].text trace['model_latency_ms'] = int((time.time() - t1) * 1000) trace['input_tokens'] = response.usage.input_tokens trace['output_tokens'] = response.usage.output_tokens # 6. Output guardrail (stub: real check would use classifier) if len(answer) < 5: # example: reject empty responses answer = 'I was unable to generate a response. Please try again.' # 7. Cache the response CACHE.append((q_emb, answer)) trace['total_latency_ms'] = int((time.time() - t0) * 1000) return {'response': answer, 'trace': trace}
Real-world usage
-
Notion AI (Chip Huyen, AI Engineering Ch.10): Uses a context pipeline that assembles page context, user writing history, and workspace structure before every model call. The context quality improvement — not model size — drove the biggest quality gains.
-
Intercom's Fin (AI support agent): Multi-tier model routing — simple FAQ queries go to a small cached model; complex cases escalate to a larger model; unresolved cases route to a human agent. Cache hit rate ~35% on FAQ traffic.
-
Cursor (AI coding assistant): Uses semantic caching for common coding patterns, a model router between Sonnet (everyday edits) and Opus (architecture questions), and an agent loop with hard caps (max 10 tool calls per request).
-
Customer support AI (generic pattern): Input guardrail blocks PII/profanity → semantic cache handles repeat queries (30-40% hit rate) → RAG retrieves knowledge base → output guardrail checks citations → trace emits to observability dashboard.
-
Chip Huyen (Ch.10) on feedback systems: 'The most valuable signal is often the next user message. If the user immediately re-asks the same question differently, the previous response failed. This implicit signal, aggregated at scale, is more reliable than explicit thumbs ratings, which suffer from response bias.'
Trade-offs
Guardrail coverage vs. latency: each guardrail adds 10-100ms. A system with 5 guardrails adds 50-500ms before the model call. Choose guardrails that address real, observed failure modes — not hypothetical risks. Start with the highest-ROI guardrail (usually input safety) and add others only when a specific failure justifies the latency cost.
Semantic cache threshold vs. quality: a high threshold (0.97) reduces false cache hits but also reduces hit rate. A low threshold (0.85) increases hit rate but risks returning a cached answer to a superficially similar but semantically different question. Calibrate the threshold using human evaluation on a sample of cache hit pairs.
Model routing accuracy vs. cost: a more accurate router (fine-tuned classifier) costs more to build and adds latency. A simple rule-based router is cheap and fast but miscategorizes edge cases. For most systems: start with rules, measure misrouting rate from traces, add a classifier only if misrouting is causing measurable quality problems.
Agent autonomy vs. safety: more tool calls = more capability but more blast radius. Err toward human-in-the-loop for any irreversible action (send email, delete, charge) and autonomous execution for reversible reads. The cost of a mis-sent email is higher than the UX friction of one confirmation click.
Visual explanation
End-to-End Production AI Request Path:
User Request │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ INPUT GUARDRAIL │ │ • PII detection / redaction │ │ • Prompt injection detection │ │ • Topic / policy check │ │ • Rate limiting │ └─────────────────────┬───────────────────────────────────────┘ │ safe ▼ ┌─────────────────────────────────────────────────────────────┐ │ SEMANTIC CACHE │ │ • Embed query → search vector cache │ │ • If similarity > threshold: return cached response │ │ • Cache hit rate typically 20-40% for product assistants │ └─────────────────────┬───────────────────────────────────────┘ │ cache miss ▼ ┌─────────────────────────────────────────────────────────────┐ │ CONTEXT CONSTRUCTION │ │ • RAG retrieval (vector search + rerank) │ │ • Memory lookup (user profile, session history) │ │ • Tool pre-fetch (if predictable tool needed) │ │ • Context budget management (fit within window) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ MODEL ROUTER / GATEWAY │ │ • Classify task complexity → select model tier │ │ • Tier 1: fast/cheap (Haiku, GPT-4o-mini) for simple tasks│ │ • Tier 2: capable (Sonnet, GPT-4o) for reasoning tasks │ │ • Tier 3: powerful (Opus, o3) for research/analysis │ │ • Load balance, retry, fallback │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ MODEL CALL + AGENT LOOP (if agentic) │ │ • Streaming response with tool use │ │ • Tool calls dispatched concurrently where independent │ │ • Max iterations / time / cost budget enforced │ │ • Interrupt / human-in-the-loop for high-risk actions │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ OUTPUT GUARDRAIL │ │ • Hallucination / citation check │ │ • PII in output (re-redact before delivery) │ │ • Safety / policy compliance │ │ • Format validation (JSON schema, length) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ OBSERVABILITY + FEEDBACK │ │ • Emit trace (input, context, output, latency, cost) │ │ • Async eval (LLM-as-judge, retrieval precision) │ │ • Capture implicit feedback (thumbs, copy, re-ask signals) │ │ • Write to evaluation dataset for future fine-tuning │ └─────────────────────────────────────────────────────────────┘ │ ▼ Response to User
Advantages
- —
Layered architecture allows each component to be improved independently — swap the reranker without touching the guardrail
- —
Semantic caching can reduce cost and latency by 30-40% for high-repetition use cases without sacrificing quality
- —
Model routing dramatically reduces cost: routing 70% of traffic to a Tier 1 model (10x cheaper) cuts total model spend by ~63%
- —
Guardrails catch failures before they reach users — input guardrails are cheap (milliseconds) and prevent expensive model calls for blocked requests
- —
Observability + feedback loop turns a static deployment into a continuously improving system
Disadvantages
- —
Each component adds latency — a fully layered architecture can add 100-500ms to the critical path if not implemented with async and caching
- —
Semantic cache staleness: cached responses can become outdated when the underlying knowledge base changes
- —
Model router complexity: the router itself can be wrong — routing a complex query to Tier 1 produces a poor answer that damages user trust
- —
Guardrail over-triggering: input classifiers can block legitimate requests (false positives), creating user friction
- —
Feedback loop lag: traces and evaluations from today inform improvements deployed next week — the system improves slowly unless the feedback loop is automated
Common mistakes
- —
Building guardrails after launch, not before. Input and output guardrails are much cheaper to add before first deployment than to retrofit after a safety incident. Even a basic blocklist and topic classifier on day 1 prevents the most common failure modes.
- —
Caching everything indiscriminately. Personalized responses, time-sensitive answers, and queries referencing user state must never be served from a semantic cache shared across users. Always segment the cache by query type and validate that cached responses are user-agnostic.
- —
No hard cap on agent loops. Without a max-iterations or time-budget constraint, a misconfigured tool (or an adversarial prompt) can cause an agent to spin indefinitely, burning tokens and potentially taking unintended actions. Always set both a step limit and a wall-clock timeout.
- —
Treating feedback as ground truth. Explicit thumbs ratings are useful signals but are noisy and biased (positive skew from politeness, negative skew from frustration). Never train directly on raw feedback without human review of a sample. Treat it as a signal for prioritization, not as a training label.
- —
Skipping context budget management. If the retrieved documents plus conversation history exceed the context window, the system truncates from the end — which silently drops the system prompt, instructions, or the most relevant documents. Explicitly manage the budget: system prompt first, most relevant documents next, history last (and summarize it if too long).
🎤 Interview questions
Walk me through the request path of a production AI assistant from user input to response. What components do you include and why?
How does semantic caching differ from traditional key-value caching, and what are the risks of setting the similarity threshold too low?
You're building a model router for a customer support AI. What signals would you use to decide between a cheap fast model and an expensive reasoning model? How would you evaluate your router's accuracy?
How do you design a user feedback system for an AI product? What are the limitations of explicit feedback (thumbs up/down) vs. implicit signals?
📂 Subtopics
Context Construction, Guardrails, and the Model Gateway
The three highest-leverage components in any production AI system: assembling context before the model call, filtering inputs and outputs, and routing requests to the right model at the right cost.
~50 min
Semantic Caching and Agent Orchestration
How semantic caching cuts latency and cost for repetitive queries, and how to run agent loops in production with appropriate blast-radius controls.
~40 min
Observability, Evaluation, and User Feedback Systems
How to make an AI system observable and improvable: structured traces, async evaluation, and extracting quality signals from explicit and implicit user feedback.
~40 min