Output Guardrails: Content Filtering, Toxicity Detection, and PII Redaction

The previous two subtopics covered risks on the INPUT side — attackers trying to manipulate what the model does. This subtopic covers the complementary OUTPUT side: even without any attack at all, a model can occasionally generate content that's unwanted, unsafe, or leaks sensitive information, simply as a byproduct of imperfect training or an unlucky generation. Output guardrails catch this AFTER the model generates a response but BEFORE that response reaches the user.

Content filtering checks generated text against disallowed categories (violence, harassment, illegal activity, and similar policy-defined categories) before it's returned. In practice this is usually implemented as a SEPARATE, smaller, fast classifier model running alongside the main LLM — specifically because running a dedicated classifier is far cheaper and faster than asking the main LLM to self-critique its own output, and because a smaller model trained SPECIFICALLY for classification tends to be more consistent at this narrow task than a general-purpose LLM doing it as a secondary instruction.

Toxicity detection is a more specific case of content filtering, scoring text along dimensions like insult, profanity, threat, or identity-based attack — often as a continuous score (e.g. 0 to 1) rather than a strict binary allow/block, which lets an application set its own threshold appropriate to its context (a children's education app needs a stricter threshold than an internal engineering tool). Open tools like Perspective API (Google/Jigsaw) and various open-source toxicity classifiers are commonly used for this specific sub-problem.

PII (Personally Identifiable Information) redaction addresses a different concern: even benign, non-toxic output can be a genuine problem if it accidentally includes information that shouldn't be exposed — a model summarizing internal documents might inadvertently include someone's home address or a customer's account number that appeared in the source material, or in rare cases, might have memorized and regurgitated real personal data from its training set. PII redaction scans generated output for patterns matching emails, phone numbers, social security numbers, credit card numbers, and similar structured sensitive data, and either masks it (replacing with '[REDACTED]' or similar) or blocks the response entirely, depending on the application's risk tolerance.

The unifying design principle across all three: output guardrails work as a POST-GENERATION checkpoint, independent of whatever the input-side defenses (previous two subtopics) did or didn't catch — even if a jailbreak or injection succeeds in getting the model to attempt something disallowed, a well-designed output guardrail still has a chance to catch the RESULT before a user ever sees it, which is exactly why defense-in-depth (the next subtopic's framing) treats output filtering as a distinct, necessary layer rather than a redundant one.

# A toy pipeline: toxicity scoring (threshold-based) + PII pattern
# redaction, applied to a model's OUTPUT before it reaches a user.
import re

def toxicity_score(text: str) -> float:
    """Stand-in for a real toxicity classifier (e.g. Perspective API) --
    a real system uses a trained model, not a keyword count."""
    flagged_terms = ["idiot", "stupid", "hate you"]
    hits = sum(1 for term in flagged_terms if term in text.lower())
    return min(hits * 0.4, 1.0)

PII_PATTERNS = {
    "email": r"[\w.+-]+@[\w-]+\.[\w.-]+",
    "phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
    "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
}

def redact_pii(text: str) -> tuple[str, list[str]]:
    """Scan and mask structured sensitive data before returning output."""
    redacted_types = []
    for pii_type, pattern in PII_PATTERNS.items():
        if re.search(pattern, text):
            redacted_types.append(pii_type)
            text = re.sub(pattern, f"[REDACTED_{pii_type.upper()}]", text)
    return text, redacted_types

def output_guardrail_pipeline(model_output: str, toxicity_threshold: float = 0.5) -> dict:
    """The post-generation checkpoint: run BOTH checks before returning
    anything to the user."""
    score = toxicity_score(model_output)
    if score >= toxicity_threshold:
        return {"status": "blocked", "reason": f"toxicity score {score:.2f} exceeds threshold"}
    redacted_text, redacted_types = redact_pii(model_output)
    return {"status": "allowed", "output": redacted_text, "pii_redacted": redacted_types}

example_1 = "Contact our support team at jane.doe@example.com or call 555-123-4567."
example_2 = "You're an idiot for asking that."

print(output_guardrail_pipeline(example_1))
print(output_guardrail_pipeline(example_2))