Handling JSON Failures: When LLMs Break Format and How to Recover
~15 min read
Even with good prompting, JSON output can still break — trailing commentary, truncated output, or malformed syntax. Retry-with-error, lenient parsing, and validation-with-repair are the standard recovery patterns.
Handling JSON Failures: When LLMs Break Format and How to Recover is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Trailing commentary after the JSON is the most common failure — fix with stop sequences or lenient extraction (find the {...} block, don't assume the whole response is JSON)
- •Truncation from a too-low max_tokens cuts JSON off mid-object — size max_tokens generously and check for it at runtime
- •Syntactically valid JSON can still violate your actual schema (wrong types, missing fields) — catch this with Pydantic (or similar) validation, not just json.loads()
- •The standard recovery pattern: retry with the concrete parse/validation error fed back to the model, not a blind retry
- •Models self-correct far more reliably when given a specific error message than when just asked to try again blind