LLM Output Safety in Production: Reducing False Positives with Tiered Review and Gray Thresholds

The Problem

Many teams prioritize performance, cost, and latency when integrating large language models, leaving content safety as an afterthought. This sequence often leads to two issues: first, discovering post-deployment that some responses are unsuitable for display; second, tightening safety policies so aggressively that legitimate user requests are falsely blocked, degrading the user experience.

Content safety is not a simple boolean “is it violating?” check placed in front of an API. The real production challenge involves deciding: which content must be blocked, which can be downgraded, which should be routed for human review, and which only needs audit logging. The system must also account for input risk, output risk, user age, business type, regional compliance, and historical behavior.

OpenAI’s Moderation documentation explains that moderation models can detect harmful content in text and images, with results used for filtering, routing to human review, or intervening on accounts submitting flagged content. Azure AI Content Safety similarly emphasizes detection capabilities for user-generated and AI-generated content in applications. AWS Bedrock Guardrails and Google Model Armor go further, integrating content filtering, sensitive information protection, input/output blocking, auditing, and policy enforcement into the generative AI application protection pipeline.

This article does not discuss “how to write a perfect safety prompt.” Instead, it focuses on more practical content safety engineering for production systems: tiered review, gray thresholds, false positive mitigation, human review, and deployment checklists.

Core Principles

1. Move from Binary Classification to Risk Tiering

The most common mistake is compressing content safety results into a single field: blocked: true | false. While simple to implement, this approach hinders effective governance.

Production systems benefit from retaining three types of information:

• category: The risk category hit, e.g., harassment, hate, adult, violence, self-harm, illegal advice, sensitive information.
• severity: The risk level, e.g., low, medium, high, critical.
• confidence or score: The classifier’s confidence in its judgment.

This decouples “detection” from “action.” The detection model only identifies potential risks; the business policy determines the action. Low-risk content might prompt the user to rephrase; medium-risk content can be rewritten or partially masked; high-risk content can be rejected; borderline cases can enter human review.

Risk Level	Confidence Range	Suggested Action
low	< 0.40	Log only, return normally
medium	0.40 - 0.70	Safe rewrite or partial masking
high	0.70 - 0.90	Route to human review queue
critical	> 0.90	Block directly

2. Inspect Both Input and Output

Checking only user input is insufficient, as model output can introduce new risks. For example, a user might make a borderline request that is not obviously violating, but the model’s completion expands into details unsuitable for display. Google Model Armor’s architecture explicitly includes two checkpoints: first, inspect the prompt; second, inspect the model-generated response; if needed, sanitize either before proceeding.

We recommend splitting the pipeline into four inspection points:

1. pre-input check: Before user input enters the model.
2. context check: After the system assembles historical context, tool results, and retrieved content.
3. post-output check: Before the model’s response is returned to the user.
4. feedback check: After user reports, human reviews, and appeals.

The third point is most often overlooked but is critical for production systems.

3. Apply Thresholds by Scenario with Gray Releases, Not Globally

Content safety classifiers produce false positives. OpenAI’s documentation also notes that the moderation endpoint’s underlying models are continuously updated, so custom policies relying on category_scores may need recalibration over time. This means thresholds should not be hardcoded, nor should all business scenarios share a single global threshold.

A more robust approach is to build a policy matrix:

policy_version: content-safety-2026-07-03
rules:
  - scene: public_chat
    category: violence
    threshold:
      block: 0.86
      review: 0.62
      log_only: 0.40

  - scene: customer_support
    category: self_harm
    threshold:
      block: 0.92
      review: 0.55
      safe_completion: 0.30

  - scene: internal_knowledge_base
    category: pii
    threshold:
      redact: 0.50
      review: 0.75

The same risk category can have different actions across business scenarios. For example, public community content should be more conservative, internal knowledge base Q&A can prioritize redaction and auditing, while customer support scenarios may need to route some high-risk conversations to human agents.

Engineering Implementation

1. Recommended Architecture

A production-grade content safety pipeline typically includes the following modules:

• Policy Service: Stores policy versions, thresholds, scenarios, actions, and gray release ratios.
• Moderation Adapter: Integrates with OpenAI Moderation, Azure AI Content Safety, Bedrock Guardrails, Model Armor, or custom classifiers.
• Decision Engine: Maps classification results to actions like allow, block, redact, rewrite, review, log_only.
• Review Queue: Handles borderline samples, user appeals, and human review results.
• Audit Store: Records policy version, model version, classification scores, executed actions, and final outcomes.
• Metrics Dashboard: Monitors block rate, false positive rate, review pass rate, user appeal rate, and category trends.

The key point is: business services should not hardcode classifier thresholds. They only pass the scenario, user type, input, output, and context summary to the policy service, which returns the final action.

2. Decision Engine Example

Below is a simplified TypeScript pseudocode illustrating how detection results are translated into actions.

type ModerationResult = {
  category: string;
  score: number;
  severity?: "low" | "medium" | "high" | "critical";
};

type SafetyAction = "allow" | "log_only" | "redact" | "review" | "block";

type PolicyRule = {
  scene: string;
  category: string;
  logOnlyScore: number;
  reviewScore: number;
  blockScore: number;
  redactScore?: number;
};

function decideAction(
  scene: string,
  results: ModerationResult[],
  rules: PolicyRule[]
): SafetyAction {
  let finalAction: SafetyAction = "allow";
  for (const result of results) {
    const rule = rules.find(
      r => r.scene === scene && r.category === result.category
    );
    if (!rule) continue;
    if (result.score >= rule.blockScore) return "block";
    if (result.score >= rule.reviewScore)
      finalAction = maxAction(finalAction, "review");
    else if (rule.redactScore && result.score >= rule.redactScore)
      finalAction = maxAction(finalAction, "redact");
    else if (result.score >= rule.logOnlyScore)
      finalAction = maxAction(finalAction, "log_only");
  }
  return finalAction;
}

function maxAction(a: SafetyAction, b: SafetyAction): SafetyAction {
  const order: SafetyAction[] = ["allow", "log_only", "redact", "review", "block"];
  return order.indexOf(a) >= order.indexOf(b) ? a : b;
}

This code is just a skeleton. A real system must also consider user age, regional policies, product lines, request sources, model types, historical violation counts, and whether the request hits a sensitive business flow.

3. Audit Logs Must Record Policy Version

When troubleshooting content safety issues, teams often only see that “a message was blocked” without knowing why. We recommend recording at least the following for every decision:

Field	Description
request_id / conversation_id	Request or conversation identifier
policy_version	Current effective policy version
moderation_provider	Moderation service provider
moderation_model	Moderation model version
input_check_result	Input inspection result
output_check_result	Output inspection result
action	Final executed action
threshold_snapshot	Threshold snapshot at decision time
reviewer_result	Human review result
appeal_result	User appeal result
created_at	Timestamp

policy_version and threshold_snapshot are particularly important. Without them, it’s difficult to later answer questions like “Why was this user blocked at that time?”, “Did a policy upgrade cause the block rate to spike?”, or “Should the old policy version be rolled back?“

4. Human Review is Not Just a Safety Net; It’s a Training Data Source

The human review queue should not only serve as a customer support remedy. Its more important value is providing continuous calibration data.

We recommend categorizing review samples into four types:

1. false positive: Incorrectly blocked; should lower the threshold or add whitelist context.
2. false negative: Incorrectly allowed; should tighten the threshold or add rules.
3. policy ambiguous: Policy unclear; needs business and legal clarification.
4. model uncertain: Classifier score fluctuates; needs multi-model cross-validation or human priority.

Such labeled data can feed into a safety regression set. However, do not mix it with general model evaluation. The content safety regression set should be stratified by risk category, language, region, scenario, and user group.

Applicable Scenarios

Content safety gateways are most suitable for:

• Public-facing AI chat, search, Q&A, or writing products.
• Enterprise customer support, finance, healthcare, education, and other scenarios sensitive to output boundaries.
• Systems that allow users to upload images, documents, web pages, or long texts for model-generated summaries.
• Multi-model platforms or internal LLM platforms requiring unified content safety policies.
• Applications with human operations, appeals, auditing, or compliance requirements.

For internal R&D testing tools, at least retain a log_only mode. Google Model Armor’s “Inspect only” mode is ideal for gray releases: first record potential violations and hit distributions without blocking traffic, then switch to active blocking once the team understands the false positive rate.

Common Misconceptions

Misconception 1: Integrating a Content Safety API Completes Governance

APIs only provide detection results; they cannot define your business policy. The real challenge lies in deciding: which content to allow, which to rewrite, which to route to human review, and which to block.

Misconception 2: Stricter Thresholds Mean Safer Systems

Overly strict thresholds generate a high volume of false positives. False positives cause users to bypass the system, rephrase their requests, complain, or even fail to complete legitimate tasks. Content safety is not about maximizing block rates; it’s about balancing risk, user experience, and human review costs.

Misconception 3: One Set of Rules Fits All Scenarios

Public communities, internal knowledge bases, customer support tickets, children’s products, financial advice, and code assistants have different risk boundaries. Uniform rules tend to over-block in low-risk scenarios and under-block in high-risk ones.

Misconception 4: Ignoring Multilingual and Implicit Expressions

Content safety classifiers may perform inconsistently across languages, dialects, metaphors, and group-specific expressions. Public research also indicates that commercial content moderation APIs can simultaneously over-moderate and under-moderate. Therefore, prepare test sets covering multiple languages, regions, and expression styles before deployment.

Deployment Checklist

• Are user input, assembled context, and model output all inspected?
• Are detection results and execution policies decoupled?
• Are policy_version, threshold_snapshot, and moderation_model recorded?
• Are multiple actions supported: log_only, review, redact, block?
• Can thresholds be configured by business scenario, region, user type, and product line?
• Are human review and appeal processes established?
• Are false positive rate, false negative rate, review pass rate, and user complaint rate tracked?
• Are policy gray releases, rollbacks, and audit exports supported?
• Are boundaries defined for images, documents, tool responses, and multi-turn context?
• Is it clear what content is recorded, for how long, and who can access it?

FAQ

Should content safety be placed at the gateway layer or the business service layer?

It is best placed in a unified policy or platform layer, but allow business services to pass scenario parameters. Placing it entirely in business services leads to fragmented rules; placing it entirely in the gateway may lose business context. A better approach is for the platform to provide unified detection, auditing, and policy execution, while the business side provides scenario, user type, and risk boundaries.

Should a hard error be returned when output content is blocked?

Not always. Depending on the risk level, you can choose safe rewriting, partial masking, explanatory rejection, routing to human review, regeneration, or simply prompting the user to modify their input. Returning a hard error is simple but often harms the user experience and reduces the collection of reviewable samples.

References

1. OpenAI Moderation Guide: https://developers.openai.com/api/docs/guides/moderation
2. OpenAI Safety Best Practices: https://developers.openai.com/api/docs/guides/safety-best-practices
3. Azure AI Content Safety Overview: https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview
4. Amazon Bedrock Guardrails: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html
5. Google Cloud Model Armor Overview: https://docs.cloud.google.com/model-armor/overview
6. BingoGuard: LLM Content Moderation Tools with Risk Levels: https://arxiv.org/abs/2503.06550
7. Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations: https://arxiv.org/abs/2503.01623