Semantic Caching and Agent Orchestration

Semantic Caching

Traditional caching requires exact key matches — useless for natural language where users phrase the same question differently. Semantic caching stores (query_embedding, response) pairs and returns a cached response when a new query's embedding is close enough to a cached one.

# Semantic cache lookup
def semantic_cache_get(query: str, cache: list, threshold=0.92):
    q_emb = embed(query)  # embed incoming query
    for cached_emb, cached_resp, cached_ts in cache:
        sim = cosine_similarity(q_emb, cached_emb)
        if sim >= threshold:
            return cached_resp  # cache hit
    return None  # cache miss

Typical cache hit rates in production: 20-40% for FAQ/support assistants, 5-15% for open-ended chat. Each hit saves the full model call cost and latency.

Cache design decisions:

• Threshold: 0.92-0.95 is a safe starting point; calibrate by human-reviewing 50 hit pairs
• Scope: never share cache across users for personalized responses
• TTL: set expiry based on how often the underlying knowledge changes (1h for news, 24h for FAQ, 7d for product docs)
• Invalidation: clear relevant cache entries when the knowledge base is updated
• Storage: Redis (fast in-memory) + pgvector or FAISS (vector index)

Agent Orchestration in Production

Agents that call tools in a loop are powerful but risky in production:

Risk 1 — Blast radius: a tool that sends emails or charges a card can cause real harm if triggered unexpectedly Risk 2 — Cost runaway: uncapped loops can consume large token and API call budgets Risk 3 — Latency variance: tool call chains are unpredictable; P99 latency can be 10x P50

Production controls:

MAX_STEPS = 20          # hard cap on tool call iterations
MAX_WALL_CLOCK_S = 30   # abort after 30 seconds regardless
MAX_COST_USD = 0.50     # per-session spend limit
CONFIRM_BEFORE = [      # irreversible actions require human approval
    'send_email', 'delete_record', 'charge_payment'
]

Concurrency: independent tool calls should run in parallel:

# If model requests both 'get_user_profile' and 'get_order_history',
# run them concurrently — don't wait for one to finish before starting the other.
import asyncio
results = await asyncio.gather(
    get_user_profile(user_id),
    get_order_history(user_id),
)

Tool permission tiers:

• Tier A (read-only): autonomous execution always allowed
• Tier B (write, reversible): execute with audit log
• Tier C (write, irreversible): require human confirmation before execution