Handling JSON Failures: When LLMs Break Format and How to Recover

Even with a well-specified schema, few-shot examples, and native JSON mode enabled, JSON output can still break in practice — especially with models or configurations that don't support a hard structural JSON mode, or with genuinely tricky inputs. Knowing the common failure modes and how to recover from each is what separates a demo pipeline from a production one.

The most common failure is trailing commentary — the model emits valid JSON but then adds a sentence like 'Let me know if you need anything else!' after the closing brace. If you're doing a naive json.loads() on the raw response, this breaks parsing entirely even though the actual JSON was fine. The fix is either a stop sequence right after the expected closing brace (see the max-tokens/stop-sequences subtopic), or a lenient extraction step that finds the first complete JSON object in the response text rather than assuming the entire response IS the JSON.

A second common failure is truncation — if max_tokens is set too low for the actual output size, the JSON gets cut off mid-object and becomes syntactically invalid. This one is straightforward to prevent (size max_tokens generously) but needs a runtime check to catch when it still happens, since a truncated response won't raise an error on its own until you actually try to parse it.

A third failure mode is subtler: syntactically valid JSON that violates your actual schema — a string where you expected a number, a missing required field, or a value outside your enum's allowed set. This passes json.loads() cleanly but fails your business logic downstream. The standard fix is schema validation (e.g. with Pydantic) immediately after parsing, which catches these cases with a clear error rather than a confusing failure somewhere else in your pipeline.

The general recovery pattern for all three: wrap the parse-and-validate step in a retry loop that, on failure, sends the error back to the model along with the original request ('Your last response failed to parse: . Please fix and resend valid JSON matching the schema.') — LLMs are often able to self-correct given a specific, concrete error message, far more reliably than they get it right blind on a bare retry.

import json
import re
from pydantic import BaseModel, ValidationError
from openai import OpenAI

client = OpenAI()

class EmailExtraction(BaseModel):
    sender_intent: str
    key_points: list[str]
    priority: str  # "low" | "medium" | "high"

def extract_json_object(text: str) -> str:
    """Lenient extraction: find the first {...} block, ignoring trailing prose."""
    match = re.search(r"\{.*\}", text, re.DOTALL)
    if not match:
        raise ValueError("No JSON object found in response")
    return match.group(0)

def extract_with_retry(prompt: str, max_retries: int = 2) -> EmailExtraction:
    messages = [{"role": "user", "content": prompt}]
    for attempt in range(max_retries + 1):
        resp = client.chat.completions.create(model="gpt-4.1", messages=messages)
        raw = resp.choices[0].message.content
        try:
            parsed = json.loads(extract_json_object(raw))
            return EmailExtraction(**parsed)  # schema validation
        except (ValueError, json.JSONDecodeError, ValidationError) as e:
            if attempt == max_retries:
                raise
            # Feed the concrete error back — models self-correct well from this
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content": f"That failed to parse: {e}. Resend valid JSON."})