Article

Structured Outputs in Production: Stabilizing JSON Schema Generation with Constrained Decoding

A systematic guide to Structured Outputs and Constrained Decoding: core principles, schema design, server integration, performance, error handling, and common pitfalls for reliable LLM JSON generation.

Background: Why “Please Output JSON” Isn’t Enough in Production

When large language models enter business systems, their output is no longer just natural language for human reading. Customer service ticket routing, contract field extraction, Agent tool calls, report generation, risk control explanations, and RAG answer citations all require the model to output structured data that programs can reliably parse.

Early approaches typically relied on prompt constraints:

Please output strictly in JSON, no Markdown, no explanation.

Such prompts work in demos, but in production they lead to several typical issues:

Issue TypeManifestationImpact
Markdown wrappingModel wraps JSON in a ```json code blockDownstream parsing fails
Field name driftriskLevel becomes risk_levelDTO deserialization error
Type errorsRequired fields missing, numbers output as stringsSchema validation fails
Enum drifthigh becomes urgentBusiness logic branches break
Unstable tool parametersAgent parameters seem reasonable but don’t match contractTool call fails

Structured Outputs doesn’t solve “making the model better at writing JSON”; it elevates output format from a “prompt suggestion” to a “decoding-stage constraint.” That’s the value of Constrained Decoding: at each token generation step, based on JSON Schema, regex, EBNF, or context-free grammar, only tokens that are legal in the current state are allowed into the candidate set.

Core Principle: From Probabilistic Generation to Constrained Generation

The normal generation process of an LLM can be understood as: the model gives a probability distribution for the next token based on context, then samples or selects a token. Regular prompts can only influence probabilities, not fundamentally prohibit illegal structures.

Constrained Decoding adds a mask layer after the model’s logits but before sampling:

  1. Compile the business schema into an executable grammar state machine or parser.
  2. At each generation step, determine which tokens are legal for the next step based on the current generated content.
  3. Mask out the probabilities of illegal tokens.
  4. Continue sampling or selecting only from the legal token set.
  5. Continue until the complete structure is generated or a stop condition is triggered.

This means that if the schema requires the field priority to be only low, medium, or high, even if the model tends to output urgent, the decoder will exclude that path. Structural correctness no longer depends entirely on the model “obeying.”

But this also brings an easily overlooked fact: Constrained decoding guarantees structure, not factual or business judgment correctness. The model can stably output:

{ "priority": "high", "reason": "The user expressed an urgent request", "confidence": 0.82 }

This JSON may perfectly match the schema, but whether priority is correctly judged still requires business rules, human-annotated datasets, offline evaluation, or online spot-checking.

Comparison of Three Modes: Structured Outputs, JSON Mode, and Function Calling

JSON Mode

The goal of JSON mode is usually to make the output valid JSON. It reduces obvious formatting errors but does not guarantee that the field set, field types, nested structures, and enum values match your schema.

DimensionJSON ModeStructured Outputs
Constraint levelOutput valid JSONOutput strictly matches Schema
Field guaranteeNot guaranteedGuarantees required fields exist
Type guaranteeNot guaranteedGuarantees type matching
Enum constraintNot constrainedStrictly limits allowed values
Extra fieldsMay appearadditionalProperties: false can prevent
  • Suitable for: Lightweight extraction, internal scripts, non-critical pipelines.
  • Unsuitable for: Payment parameters, database writes, automatic approvals, tool call parameters, API return values requiring strong contracts.

Structured Outputs

Structured Outputs requires developers to provide a Schema and forces the model to output matching that Schema during generation. When OpenAI released Structured Outputs in August 2024, it clearly distinguished JSON mode from strict Schema following: JSON mode only increases the probability of valid JSON; Structured Outputs is the true constraint for developers targeting a JSON Schema.

  • Suitable for: Structured extraction, front-end/back-end contracts, LLM-generated business objects, Agent intermediate states, complex form generation.

Function Calling / Tool Calling

Function Calling focuses more on “the model deciding which tool to call and what parameters to pass.” When tool parameters enable strict Schema, it is essentially a major use case of Structured Outputs.

  • Suitable for: Agent tool calls, search, database queries, order queries, CRM updates, parameter validation before code execution.

In production, don’t treat Structured Outputs as a single API parameter. Instead, design it as a three-layer capability.

Layer 1: Schema Contract

The Schema is a business contract, not part of the prompt. It should be version-controlled and aligned with downstream service DTOs, database write structures, or event formats.

A ticket routing schema could be designed like this:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["category", "priority", "summary", "confidence"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account", "other"]
    },
    "priority": {
      "type": "string",
      "enum": ["low", "medium", "high"]
    },
    "summary": {
      "type": "string",
      "description": "A concise summary of the customer issue."
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    }
  }
}

Schema design should follow four principles:

  • Few and stable fields: Don’t cram all potential fields in at once.
  • Clear enums: Fields that need branching logic on the backend should use enums first.
  • Disallow extra fields: Use additionalProperties: false for critical pipelines.
  • Evolvable versions: Have a version strategy for adding fields, deprecating fields, and extending enums.

Layer 2: Generation Adapter

Different inference platforms have different interfaces. The parameters and support ranges of OpenAI, Claude, vLLM, SGLang, TensorRT-LLM, XGrammar, Guidance, and Outlines are not identical. Therefore, it’s recommended to add an adapter layer in your business code that converts the business Schema into the format supported by the target backend.

type StructuredBackend = "openai" | "vllm" | "sglang";

interface StructuredRequest {
  model: string;
  messages: Array<{ role: string; content: string }>;
  schemaName: string;
  schema: Record<string, unknown>;
  temperature?: number;
}

async function generateStructured(
  backend: StructuredBackend,
  request: StructuredRequest
): Promise<unknown> {
  if (backend === "openai") {
    return callOpenAIStructuredOutputs(request);
  }
  if (backend === "vllm") {
    return callOpenAICompatibleServer(request, {
      structured_outputs: { json: request.schema }
    });
  }
  if (backend === "sglang") {
    return callOpenAICompatibleServer(request, {
      response_format: {
        type: "json_schema",
        json_schema: request.schema
      }
    });
  }
  throw new Error(`Unsupported backend: ${backend}`);
}

The value of this adapter layer is that business code only cares about “I want a TicketTriageResult,” not whether the underlying system is OpenAI’s native Structured Outputs, vLLM’s structured_outputs, or SGLang’s json_schema.

Layer 3: Validation & Recovery

Even with constrained decoding, post-validation must be retained. Reasons include:

  • The model may produce incomplete output due to refusal, length limits, or server errors.
  • Different platforms support different subsets of JSON Schema.
  • Some constraints only guarantee syntax, not business semantics.
  • Downstream systems still need permission, scope, and state machine validation.

Recommended flow:

const raw = await generateStructured(backend, request);
const parsed = schemaValidator.safeParse(raw);

if (!parsed.success) {
  logStructuredOutputError({ raw, errors: parsed.error });
  return fallbackToHumanReview(raw);
}

const businessCheck = validateBusinessRules(parsed.data);
if (!businessCheck.ok) {
  return fallbackToManualQueue(parsed.data, businessCheck.reason);
}

return parsed.data;

Don’t make “retry” the only fallback. For high-risk pipelines, clearly distinguish three types of failure:

  1. Structural failure: JSON is invalid, fields missing, type errors.
  2. Schema failure: Structure is valid but doesn’t match the contract.
  3. Business failure: Contract is valid but business rules reject it.

Applicable Scenarios: Which Tasks Benefit Most from Structured Outputs

ScenarioTypical TaskCore Benefit
Information ExtractionEmail, contract, chat log, ticket field extractionDownstream direct database entry, approval flow
Agent Tool CallingSearch, query, CRM operationsParameters always match server contract
RAG Citations & EvidenceAnswer + citation ID + confidence + review flagOutput is auditable and traceable
Frontend Dynamic FormsGenerate form fields, filter conditions, chart configsFrontend renders according to Schema
Batch Offline ProcessingLarge-scale structured extractionReduced parsing failures and manual fixes

1. Information Extraction

Extracting fields from emails, contracts, chat logs, and tickets is a classic use case for Structured Outputs. With stable output structure, downstream systems can directly enter databases, approval flows, or BI.

2. Agent Tool Calling

The most common point of failure for Agents is not “not knowing how to answer,” but “unstable parameters when calling tools.” Structured Outputs ensures tool parameters always match the server contract, reducing erroneous calls, injected parameters, and invalid requests.

3. RAG Citation & Evidence Structure

RAG answers can be required to output: answer, citation snippet ID, confidence, reason for uncertainty, and whether human review is needed. This is much easier to audit than letting the model freely write an explanation.

4. Frontend Dynamic Forms or UI State

When the model needs to generate form fields, filter conditions, or chart configurations, Structured Outputs allows the frontend to render according to the Schema, rather than parsing natural language.

5. Batch Offline Processing

In batch processing tasks, failed samples amplify operational costs. Strong structured output can significantly reduce parsing failures, manual fixes, and re-runs.

Common Misconceptions

Misconception 1: Structured Outputs Eliminates the Need for Prompts

Constrained decoding handles structure, but the model still needs to understand the task. Field descriptions in the Schema, system prompts, and a few examples remain important. The SGLang documentation also recommends clearly stating the expected format in the prompt to improve output quality.

Misconception 2: The More Detailed the Schema, the Better

Complex schemas increase compilation, caching, and decoding costs. The Claude documentation explicitly mentions that Structured Outputs compiles JSON Schema into a constraint grammar, and complex schemas produce larger grammars with longer compilation times.

From an engineering perspective, first keep the Schema focused on fields that “must be stable for the business.” Avoid introducing many optional fields, deep anyOf, complex regex, and long enums just for completeness.

Misconception 3: Correct Structure Equals Correct Answer

Benchmarks like JSONSchemaBench focus not only on constraint compliance but also on efficiency, coverage, and generation quality. In production, you should also monitor structural success rate, field accuracy, business hit rate, and human review rate, not just the JSON parse success rate.

Misconception 4: All Models and Backends Support the Same Features

OpenAI, Claude, vLLM, SGLang, XGrammar, and other systems all support structured generation, but the available parameters, Schema subsets, grammar backends, limitations, and performance characteristics differ. When migrating backends, you must run regression tests, not just change the base_url.

Misconception 5: Infinite Retries on Error

Retries can mitigate occasional failures, but they cannot fix poorly designed schemas, unclear field descriptions, max_tokens being too small, insufficient model capability, or conflicting business rules. High-risk scenarios should enter a human queue or a degradation flow.

Performance and Cost: Constrained Decoding is Not Free

Constrained Decoding requires calculating the set of legal tokens at each generation step. For simple JSON, the overhead is usually manageable. For complex recursive structures, deep nesting, many enums, complex regex, or dynamic tool sets, the overhead increases significantly.

XGrammar’s research direction is precisely to reduce this overhead. XGrammar reduces the cost of structured generation through pre-checking context-free tokens, persistent stacks, and co-design with inference engines. XGrammar 2, in 2026, targets dynamic structure generation for Agents, introducing dynamic dispatch, JIT compilation, and cross-grammar caching to reduce the compilation cost of dynamic tools and conditional structures.

In production, focus on monitoring these metrics:

MetricDescription
Schema compile latencyCompilation time for first use of a Schema
Time per output token (TPOT)Whether structured output increases per-token latency
Cache hit rateWhether the same Schema reuses compiled results
Decode failure reasonLength truncation, refusal, server limits, or Schema unsupported
Fallback rateProportion falling back to human review or normal JSON mode

Launch Checklist

Schema Checks

  • Each Schema has a unique name and version number.
  • Prohibit writing sensitive information like user privacy, customer names, or ID numbers into field names, enum values, or regex.
  • Set additionalProperties: false for critical pipelines.
  • All enum values have clear business meanings.
  • Schema changes have a compatibility strategy: adding fields, deprecating fields, extending enums, and renaming fields must have a migration plan.

Inference Backend Checks

  • Clearly understand the JSON Schema subset supported by the current backend.
  • Run tests with minimal Schema, complex Schema, abnormal input, empty input, and overly long input.
  • Verify the return structure during refusal, max_tokens truncation, server timeout, and content safety interception.
  • Verify structural stability at different temperature values.
  • Verify whether batch requests, streaming output, and parallel tool calling switches affect structural guarantees.

Business Validation Checks

  • Use Zod, Pydantic, JSON Schema Validator, or server-side DTO for secondary validation.
  • Distinguish between structural errors, Schema errors, and business errors.
  • Build a human-annotated evaluation set for key fields.
  • Record the model’s raw output, parsing result, validation errors, and final processing path.
  • Route low-confidence, rule-conflicting, and high-risk results to human review.

Observability and Regression Checks

  • Record structural success rate, field accuracy, human review rate, and retry rate.
  • Run offline regression sets for every Schema change.
  • Run the same set of structured output test cases for every model upgrade.
  • Perform compatibility tests across different backends to avoid migration risks like “works on OpenAI, fails on vLLM.”
  • Store failed samples as regression test cases, not just view them in logs.

A Practical Selection Guide

If you are building a SaaS or internal enterprise system, you can choose based on the following:

  • When using closed-source APIs, prioritize official Structured Outputs or strict tool calling.
  • When using self-deployed models, first evaluate the structured output capabilities of vLLM or SGLang, and pay attention to backend support like XGrammar, Guidance, Outlines, and llguidance.
  • For simple extraction tasks, start with a flat JSON Schema; don’t design complex nesting from the beginning.
  • Agent tool parameters must use strict Schema and undergo server-side validation before tool execution.
  • Do not automatically execute high-risk tasks; add human review or secondary business rule confirmation.

FAQ

What is the difference between Structured Outputs and JSON mode?

JSON mode typically only constrains the output to be valid JSON, but cannot guarantee that fields, types, enums, and nested structures strictly match your business schema. Structured Outputs converts the schema into generation constraints, limiting allowed tokens during decoding to improve structural consistency.

Does Constrained Decoding affect answer quality?

It improves format reliability but does not automatically guarantee semantic correctness. When schemas are too detailed, enums are too numerous, or field design is unreasonable, the model may produce structurally correct but semantically weak results. Validation, evaluation, and regression testing are still necessary.

Should I prioritize prompts, retries, or constrained decoding in production?

For strong-structure scenarios like stable APIs, form extraction, and tool call parameters, prioritize constrained decoding. For open-ended summarization and explanation tasks, lightweight format prompts with post-validation can suffice. Both can be combined, but don’t rely solely on retries.

Can Structured Outputs completely replace human review?

No. It solves the structural contract problem, not factual correctness, compliance judgment, or business responsibility. For high-risk scenarios like finance, healthcare, insurance, legal, and payments, even with perfectly structured output, human review, rule engines, or approval workflows should be retained.

References

FAQ

What is the difference between Structured Outputs and JSON mode?
JSON mode typically only constrains the output to be valid JSON, but cannot guarantee that fields, types, enums, and nested structures strictly match your business schema. Structured Outputs converts the schema into generation constraints, limiting allowed tokens during decoding to improve structural consistency.
Does Constrained Decoding affect answer quality?
It improves format reliability but does not automatically guarantee semantic correctness. When schemas are too detailed, enums are too numerous, or field design is unreasonable, the model may produce structurally correct but semantically weak results. Validation, evaluation, and regression testing are still necessary.
Should I prioritize prompts, retries, or constrained decoding in production?
For strong-structure scenarios like stable APIs, form extraction, and tool call parameters, prioritize constrained decoding. For open-ended summarization and explanation tasks, lightweight format prompts with post-validation can suffice. Both can be combined, but don't rely solely on retries.
Can Structured Outputs completely replace human review?
No. It solves the structural contract problem, not factual correctness, compliance judgment, or business responsibility. For high-risk scenarios like finance, healthcare, insurance, legal, and payments, even with perfectly structured output, human review, rule engines, or approval workflows should be retained.
特效配置