Cost Optimization: Token Budgets, Caching and Model Routing

LLM serving bills are driven by two things: how many tokens you process (input + output) and how expensive the model is per token. Cost optimization is mostly about attacking those two levers without hurting quality. Three techniques give the biggest returns.

Token budgeting is the discipline of not paying for tokens you don't need. Every token in the prompt AND the output costs money and latency, so you trim aggressively: cap max output tokens to what the task actually requires; compress or summarize long conversation histories instead of resending the full transcript every turn; retrieve only the top few most-relevant chunks rather than stuffing the whole knowledge base into context; and prune verbose system prompts. Because cost scales with total tokens across every request, small per-request savings multiply across millions of calls.

Caching avoids repeating work you've already done. Exact-match response caching stores the output for a given (prompt, parameters) key and returns it instantly for identical repeat requests — common for FAQ-style traffic where the same questions recur. Semantic caching goes further: it embeds the incoming query and returns a cached answer if a sufficiently similar question was asked before, catching paraphrases the exact cache misses (with a similarity threshold you tune to avoid serving a stale answer to a subtly different question). At the serving layer, prefix/KV caching reuses the computed KV cache for shared prompt prefixes (like a long common system prompt) so you don't recompute them per request. Each layer cuts both cost and latency.

Model routing by complexity acknowledges that not every request needs your most expensive model. A router inspects the incoming request and sends easy ones (simple classification, short factual lookups, formatting) to a small, cheap model, and only escalates genuinely hard ones (multi-step reasoning, nuanced generation) to a large, expensive model. Since a large fraction of real traffic is easy, routing can slash average cost per request dramatically while keeping quality high on the requests that matter. A common pattern is a cheap classifier (or even a small LLM) doing the routing decision, sometimes with a fallback that escalates if the cheap model's confidence is low.

Used together — budget the tokens, cache the repeats, route by difficulty — these routinely cut LLM bills by large multiples with little or no quality loss, which is why cost engineering is a first-class concern in production LLM systems, not an afterthought.

# Semantic cache (catches paraphrases) + complexity-based model routing.
import numpy as np

class SemanticCache:
    def __init__(self, threshold: float = 0.92):
        self.threshold = threshold
        self.entries = []  # list of (embedding, answer)

    def get(self, embedding: np.ndarray):
        for emb, answer in self.entries:
            cos = float(emb @ embedding / (np.linalg.norm(emb) * np.linalg.norm(embedding)))
            if cos >= self.threshold:      # close enough -> reuse, skip the LLM call
                return answer
        return None

    def put(self, embedding: np.ndarray, answer: str):
        self.entries.append((embedding, answer))

def route_by_complexity(request: str) -> str:
    """Send easy requests to a cheap model; escalate hard ones."""
    tokens = request.split()
    hard_signals = ("why", "prove", "step by step", "analyze", "design")
    is_hard = len(tokens) > 60 or any(s in request.lower() for s in hard_signals)
    return "gpt-4o" if is_hard else "gpt-4o-mini"   # big vs small/cheap

print(route_by_complexity("What is the capital of France?"))        # cheap model
print(route_by_complexity("Analyze why this proof fails step by step"))  # big model