LLM Content Safety in Production: Reducing Over-Moderation with Layered Detection, Threshold Gray-Scaling, and Human Review

Background: Content Safety Is Not an if flagged then block Problem

Once a large language model (LLM) application goes live, content safety quickly evolves from “integrating a Moderation API” into a production governance challenge. The reasons are straightforward: user input may contain violating requests, borderline expressions, cited materials, complaint text, news excerpts, or educational contexts. Model output can also trigger safety classifiers during refusal responses, explanations, summaries, translations, customer service replies, or tool result paraphrasing.

If the system simply blocks based on flagged=true, the outcome is often one of two extremes: either it fails to catch genuinely risky subtle expressions, or it falsely blocks a large volume of normal content. OpenAI’s Moderation documentation explicitly advises that category_scores should be used as application policy signals, not as the sole basis for automatic blocking—refusal responses or safety explanations themselves can trigger flags due to discussing harmful content.

A production-grade content safety system needs to address four key areas:

1. Where to detect: Input, output, tool results, uploaded files, and multimodal content.
2. How to interpret scores: Categories, confidence levels, thresholds, and policy versions.
3. How to reduce false positives: Gray-scaling, human review, appeals, and allowlists.
4. How to audit and evolve: Policy changes, sample replay, threshold calibration, and re-evaluation after model upgrades.

Core Principle: Decompose Moderation into Classifier, Policy, and Action

A moderation classifier is fundamentally not a business rule engine. Its job is to produce a set of risk signals for input or output, such as whether a certain risk category is hit, the confidence score for each category, the applicable input type, and possible error states. Here’s a comparison of mainstream services:

Service	Modalities	Core Mechanism	Key Feature
OpenAI Moderation	Text, Image	category_scores (0-1)	Returns input & output results synchronously in generation requests
Azure AI Content Safety	Text, Image	Multi-class harm category + severity level	Configurable severity threshold
Google Cloud Model Armor	Text	Prompt/response check + threshold template	Inspect-only / inspect-and-block modes
Meta Llama Guard 3	Text	LLM-based safety classification	Open-source, customizable, multilingual support

Production systems should decompose the pipeline into three layers:

• Classification Layer: Calls the model or safety service, outputs standardized results.
• Policy Layer: Maps classification results to actions—allow, prompt rewrite, human review, block, account rate limit, hide output, or log-only audit.
• Execution Layer: Actually changes the business flow, e.g., not displaying output, entering a review queue, returning an alternative prompt, writing audit logs, or triggering risk control rules.

The benefit of this separation is that the classification model can be upgraded, business policies can be gray-scaled, and execution actions can vary by scenario. For example, a hit on the violence category should not be handled identically for a news summary, a medical emergency, a game plot, and instructions for real-world harm.

Recommended Architecture: Three-Stage Detection for Input, Output, and Tool Results

1. Input Side: Lightweight Admission Check Before the Model

The goal of input-side detection is not to understand all context, but to catch requests that clearly should not enter the model as early as possible. A common flow:

user_input → normalize / language detect / PII precheck
  → moderation_input → policy_decision
  → allow | rewrite hint | human review | block
  → llm_request

Input detection is suitable for handling clear violations, harassment, self-harm risk, adult content, hate speech, dangerous behavior instructions, Prompt Injection, and high-risk sensitive information. For business scenarios like customer service, education, healthcare, legal, and finance, custom business categories should be added, such as “refund fraud inducement,” “policy privacy leakage,” or “bypassing risk control rules.”

However, input detection should not be overconfident. Many requests appear normal in isolation but become risky only in context; conversely, many inputs that seem sensitive are legitimate complaints, reports, news, historical discussions, academic research, or safety education. Therefore, input detection is better suited for clear risk blocking and high-risk review triage.

2. Output Side: The Last Gate Before Display

Output detection should be placed after model generation and before content display. OpenAI’s documentation mentions that moderation scores for both input and output can be obtained in a single generation request; Google Model Armor also emphasizes that both prompts and responses can be checked. Output-side detection prevents the model from generating inappropriate results in complex dialogues, long contexts, tool results, retrieved content, or multi-turn follow-ups.

An output policy cannot simply “delete on hit.” A more reasonable decision chain is:

Risk Level	Action	Description
Low Risk	Display directly	Score well below threshold
Medium Risk	Display safe rewrite / ask user intent	Potentially borderline content
High Risk	Block output, return refusal	Clearly violating
Uncertain	Enter review queue, temporarily hide	Requires human judgment

This is especially important for refusal text, risk explanations, policy descriptions, news summaries, and medical emergency advice—content that “discusses risk but is intended to be safe.” In these cases, a secondary judgment is needed, combining the model’s response type, user intent, business scenario, and threshold policy.

3. Tool Results and Uploaded Content: Don’t Just Detect Raw User Input

In agent applications, risky content often originates from tool results, not user input. For example:

• A web search returns text unsuitable for display.
• A database query reveals sensitive fields.
• A code executor outputs a key fragment.
• A file parser reads violating text or images.

OpenAI’s documentation states that tool call parameters and tool outputs entering conversation content can be covered by moderation, but tool names, tool descriptions, tool schemas, or response-format schemas are not covered.

Production systems should integrate tool results into the content safety pipeline:

tool_result → redact / truncate / classify
  → safe_summary_or_block → llm_context
  → output_moderation

This prevents the model from verbatim repeating risky content from external systems to the user and provides audit logs with evidence of “which tool, which call, and which field the risk came from.”

Threshold Governance: From Inspect-Only to Block, Not a One-Time Rollout

Thresholds are the core configuration of a content safety system. OpenAI’s category_scores are confidence signals from 0 to 1; Google Model Armor’s templates support different confidence levels and enforcement types; Azure AI Content Safety provides a layered expression of categories and severity levels. Score semantics vary between vendors, so thresholds from one platform cannot be directly copied to another.

A four-phase rollout is recommended:

Phase One: Shadow Detection

All requests are processed normally, only classification results are logged, and user experience is unchanged. The goal is to observe hit rates, category distributions, language distributions, business scenario distributions, and high-score samples in real traffic.

Phase Two: Manual Sampling Review

Sample by category and score range, e.g., 0.3-0.5, 0.5-0.7, 0.7-0.9, 0.9+. For each score range, annotate “should block,” “should allow,” “should rewrite,” or “should review.” This step reveals which categories and contexts have the most false positives.

Phase Three: Low-Impact Action Gray-Scaling

First, roll out low-impact actions—prompt user to rewrite, hide partial content, enter human review, or rate-limit new accounts—rather than permanent bans. Enable blocking only for high-confidence, high-risk categories.

Phase Four: Policy Versioning

Every threshold, category mapping, or action rule change must generate a policy version. Logs must record:

{
  "policy_version": "safety-policy-2026-07-07-01",
  "classifier": "omni-moderation-latest",
  "category": "violence",
  "score": 0.82,
  "threshold": 0.75,
  "decision": "human_review",
  "surface": "output",
  "request_id": "req_..."
}

Without policy versions, it’s impossible to explain “why it was allowed yesterday but blocked today” or to quickly roll back when false positive rates increase.

False Positive Governance: Content Safety Systems Also Need SLOs

Metrics for a content safety system cannot focus solely on “how many violating items were blocked.” At a minimum, four types of metrics must be monitored simultaneously:

Dimension	Key Metrics
Blocking Quality	Violating sample block rate, false negative rate, human review hit rate
False Positive Cost	Normal content block rate, appeal success rate, review overturn rate, key customer collateral damage
Experience Impact	Post-block conversion rate, dialogue abandonment rate, rewrite continuation rate, human handoff rate
System Stability	Classifier timeout rate, degradation rate, latency P95, policy service error rate

Safety classifiers themselves can make mistakes. The Llama Guard 3 model card explicitly warns that deploying safety models may improve safety but can also increase refusals for benign prompts; furthermore, LLM-based guard models are limited by training data, policy coverage, multilingual capabilities, and adversarial attacks. Recent moderation API audit research also indicates that commercial content moderation APIs can exhibit both over-moderation and under-moderation, particularly with group-related expressions, counter-speech, subtle hate speech, and linguistic variants.

Therefore, a production system needs a false positive review loop: blocked content enters a queue → reviewers annotate the true result → annotated results feed back into threshold calibration, rule exceptions, evaluation sets, and regression tests.

Engineering Implementation: A Runnable Policy Decision Skeleton

The following TypeScript example demonstrates how to map classification results to business actions. The focus is not on specific thresholds, but on separating classification, policy, and execution actions:

type SafetySurface = "input" | "output" | "tool_result";
type ModerationCategory =
  | "hate" | "harassment" | "self_harm" | "sexual"
  | "violence" | "illicit" | "prompt_injection" | "pii";

type ModerationSignal = {
  category: ModerationCategory;
  score: number;
  flagged: boolean;
};

type SafetyDecision = {
  action: "allow" | "rewrite" | "review" | "block";
  reason: string;
  policyVersion: string;
  auditRequired: boolean;
};

const POLICY_VERSION = "safety-policy-2026-07-07-01";

const thresholds: Record<SafetySurface, Partial<Record<ModerationCategory, number>>> = {
  input: {
    self_harm: 0.55, violence: 0.75, harassment: 0.80,
    prompt_injection: 0.65, pii: 0.60,
  },
  output: {
    self_harm: 0.45, violence: 0.70, harassment: 0.75,
    sexual: 0.70, illicit: 0.65,
  },
  tool_result: {
    pii: 0.50, illicit: 0.70, prompt_injection: 0.60, violence: 0.75,
  },
};

export function decideSafetyAction(
  surface: SafetySurface,
  signals: ModerationSignal[],
  accountRisk: "low" | "normal" | "high"
): SafetyDecision {
  const policy = thresholds[surface];
  const hits = signals
    .filter((s) => s.score >= (policy[s.category] ?? 0.95))
    .sort((a, b) => b.score - a.score);

  if (hits.length === 0) {
    return { action: "allow", reason: "no_policy_threshold_hit",
      policyVersion: POLICY_VERSION, auditRequired: false };
  }

  const top = hits[0];

  if (top.category === "self_harm" && top.score >= 0.75) {
    return { action: "review",
      reason: "high_confidence_self_harm_needs_safe_response_or_review",
      policyVersion: POLICY_VERSION, auditRequired: true };
  }

  if (surface === "output" && top.score >= 0.9) {
    return { action: "block",
      reason: `high_confidence_${top.category}_in_output`,
      policyVersion: POLICY_VERSION, auditRequired: true };
  }

  if (accountRisk === "high" && top.score >= 0.7) {
    return { action: "review",
      reason: `risk_adjusted_review_for_${top.category}`,
      policyVersion: POLICY_VERSION, auditRequired: true };
  }

  return { action: "rewrite",
    reason: `medium_confidence_${top.category}_requires_safer_response`,
    policyVersion: POLICY_VERSION, auditRequired: true };
}

This skeleton embodies three principles:

• Different detection surfaces have different thresholds—input, output, and tool results each have their own focus.
• High-risk categories do not necessarily lead to direct bans—self-harm categories trigger more cautious responses or reviews.
• The policy version is always written to the log alongside the decision—traceable and rollbackable.

Applicable Scenarios

The content safety moderation system is suitable for the following scenarios:

1. Consumer-Facing Conversational Products: Public chatbots, customer service bots, community Q&A, educational assistants, and content generation tools.
2. UGC / External Content: Applications that support user-uploaded images, documents, web links, chat histories, customer service tickets, and other external content.
3. High-Compliance Industries: Businesses involving minors, healthcare, finance, insurance, legal, recruitment, and government affairs with strong regulatory requirements.
4. Complex Agent Systems: Agent applications supporting tool calls, web search, database queries, code execution, or knowledge base retrieval.
5. Enterprise LLM Platforms: Internal platforms that need to provide a complete audit trail for operations, compliance, and security teams.

Common Misconceptions

Misconception 1: Using flagged as the Sole Judgment

flagged is suitable for quickly identifying risks, but it cannot express business context, user intent, historical behavior, or compliance differences. Production policies should simultaneously consider category, score, detection location, input type, account risk, and scenario tags.

Misconception 2: One Threshold Fits All Businesses

The same content should be handled differently in customer complaints, news summaries, academic research, entertainment creation, and products for minors. Thresholds should be split by business line, user age group, regional regulations, content type, and risk level.

Misconception 3: Only Detecting User Input, Not Model Output

Models can generate inappropriate content in long contexts, tool results, retrieved snippets, or multi-turn inductions. Only doing input detection misses a large number of output-side risks.

Misconception 4: No Human Review Loop

Without review samples, thresholds can only be adjusted by intuition. Both false positives and false negatives require samples, labels, post-mortems, and regression tests; otherwise, policy upgrades easily introduce new business damage.

Misconception 5: Ignoring Classifier Failures and Timeouts

Safety services themselves can time out or return errors. High-risk scenarios can use fail-closed, while low-risk scenarios can use fail-open plus audit; the key is that the policy must explicitly define degradation behavior, rather than letting call exceptions randomly affect user experience.

Go-Live Checklist

Before going live, at least check the following items:

• Defined detection boundaries for input, output, tool results, uploaded files, and multimodal content.
• Defined category thresholds, action mappings, and policy versions for each business scenario.
• Supports inspect-only gray-scaling to collect real hit rates without affecting users.
• Has a human review queue and can feed review results back into the evaluation set.
• Logs record request_id, user_id, surface, category, score, threshold, decision, and policy_version.
• Dashboards exist for false positive rate, review overturn rate, appeal success rate, block rate, classifier latency, and error rate.
• Prepared replay tests for classifier upgrades, threshold adjustments, and policy releases.
• Defined degradation strategies for safety service timeouts, failures, and rate limiting.
• Implemented redaction, minimal retention, and access control for sensitive logs.

Summary

LLM content safety in production is not a point solution that ends with integrating an API. It is a systems engineering challenge requiring layered detection, threshold governance, false positive closure loops, and continuous auditing. By decomposing Moderation into classification, policy, and execution layers, combined with a four-phase gray-scale rollout from shadow detection to policy versioning, and supplemented by human review and SLO monitoring, a sustainable balance between safety and user experience can be achieved.

References

1. OpenAI Moderation API Documentation
2. Azure AI Content Safety: Harm categories
3. Google Cloud Model Armor Overview
4. Meta Llama Guard 3 8B Model Card
5. Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations