Detecting Drift: Quality Degradation and Prompt Sensitivity
~12 min read
LLM apps degrade silently — inputs shift, a model version changes under you, a prompt edit backfires. Drift detection watches output quality and input distributions over time to catch slow decay before users do.
Detecting Drift: Quality Degradation and Prompt Sensitivity is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •LLM degradation is silent — no errors, HTTP 200 everywhere — so you need continuous drift detection, not just crash monitoring
- •Input drift: incoming requests shift distribution (new topics/languages/lengths); detect via embedding clusters or simple input-stat trends
- •Output/quality drift: responses worsen for similar inputs (silent model updates, backfired prompt/retrieval changes); detect by scoring sampled outputs over time
- •Prompt sensitivity: tiny prompt edits can swing outputs — version prompts and run a regression eval suite before shipping changes; pin model versions
- •Establish healthy baselines for inputs and quality scores, then alert on statistically meaningful movement away from baseline, not on single requests