Structured Outputs in Production: Making JSON Schema Truly Usable with Constrained Decoding

Background: Why “Return JSON” Still Fails in Production

In many LLM applications, model outputs are no longer just natural language for users to read; they must feed into backend workflows: creating tickets, extracting fields, generating quotes, calling tools, updating CRMs, writing to databases, or driving frontend UIs. Relying solely on a prompt instruction like “please return JSON” is often unreliable in production. Common issues include:

• Missing required fields
• Misspelled enum values
• Drifting array structures
• Extra explanatory text in the output
• Inconsistent semantics for null values
• Unparseable content under anomalous inputs

Structured Outputs aim to upgrade the goal from “the model tries to output in a format” to “the model output must satisfy structural constraints.” The OpenAI documentation describes Structured Outputs as making the model’s response follow a developer-provided JSON Schema, thereby reducing issues like missing required keys or generating invalid enums. Open-source inference systems like vLLM, SGLang, and Outlines also provide structured generation capabilities in the form of JSON Schema, regex, and grammar.

This article focuses on a specific critical point: how to implement JSON Schema + Constrained Decoding in a production system, not just as a demo-level format control.

Core Principles: What Does Constrained Decoding Actually Constrain?

1. JSON Schema is an Output Contract, Not Just a Prompt Attachment

In a production system, JSON Schema should be treated as an interface contract, not just a description in the prompt. It should fulfill at least four types of responsibilities:

Responsibility	Description
Field Structure	Type constraints for objects, arrays, strings, numbers, booleans, etc.
Business Enums	e.g., risk_level can only be low, medium, high
Required Constraints	Fields that downstream processes depend on must be explicitly declared
Version Boundaries	When fields change, callers need to know which schema version is currently in use

The OpenAI documentation also distinguishes two scenarios: when connecting to tools or system capabilities, function calling is more appropriate; when you only need the model’s response to conform to a certain structure, you can use a structured response format. This distinction is crucial in production because “tool call parameters” and “general structured results” should be governed separately.

2. Constrained Decoding Filters Illegal Tokens During Generation

Constrained decoding is not about patching the output with regex after generation. Instead, it dynamically restricts the tokens allowed for the next step during the generation process, based on a schema, regex, or grammar. The LMSYS article on compressed finite state machines explains a common implementation path: first, convert the JSON Schema into a regex or finite state machine, calculate allowed transitions and acceptable tokens at each state, and then filter illegal tokens via a logits mask/bias.

This means constrained decoding can significantly reduce “syntactically invalid” failures, but it also introduces new engineering challenges:

• The more complex the schema, the heavier the state machine or grammar processing
• Tokenization boundaries can affect decoding efficiency
• Complex enums and deeply nested objects can increase latency
• Support for the same schema may not be entirely consistent across different backends

3. Valid JSON is Not the Correct Answer

The JSONSchemaBench paper emphasizes that evaluating structured outputs should not only check constraint satisfaction but also consider efficiency, constraint coverage, and generation quality. Production systems should adopt the same mindset: format compliance is just the first layer; content correctness, business usability, and safety control are the ultimate goals.

For example, the following output is perfectly valid structurally, but could still be a wrong business judgment:

{ "risk_level": "low", "reason": "User materials are complete", "missing_fields": [] }

If the input materials are actually missing an ID number or authorization letter, the structured output will not automatically detect the factual error. It only guarantees that the fields exist, the types are correct, and the enums are valid.

Engineering Implementation: A Controllable Structured Outputs Pipeline

1. Schemas Go into a Registry, Not Scattered in Code

It is recommended to establish a lightweight Schema Registry, even if it’s just a Git directory, to record:

schemas/
  claim_triage.v1.json
  claim_triage.v2.json
  customer_intent.v1.json
  invoice_extract.v1.json

Each schema should contain at least:

• Schema name
• Version number
• Applicable scenario
• Downstream consumers
• Whether new fields are allowed
• Semantics of null and unknown
• Example inputs and outputs
• Rollback strategy

Do not let business field names change arbitrarily. Field names themselves affect model understanding, and recent research has discussed that schema key wording can act as an implicit instruction channel, influencing structured generation behavior. Therefore, field naming should be stable, clear, and testable, rather than being overly abbreviated to save a few tokens.

2. Explicit Validation is Still Needed After Constrained Generation

Even when using Structured Outputs, it is advisable to keep a post-generation validation layer. There are three reasons for this:

1. Different models and inference backends have varying levels of support for schema features
2. Schema compliance does not equal business rule compliance
3. Situations like refusal to answer, safety filtering, timeouts, and truncation still need unified handling

A typical flow looks like this:

User Input → Prompt Builder → Schema Selection → LLM Structured Generation
→ JSON Parser → Schema Validator → Business Rule Validator
→ Typed Domain Object → Downstream Workflow

Post-generation validation should not just return “failure.” It should output observable error types:

{
  "schema_name": "claim_triage",
  "schema_version": "v1",
  "status": "invalid_business_rule",
  "error_code": "MISSING_REQUIRED_EVIDENCE",
  "field": "evidence_list",
  "retryable": false
}

3. Prompt and Schema Should Have Clear Division of Labor

A common mistake is to cram all business instructions into the schema description. A more robust division of labor is:

Component	Responsibility
Prompt	Explain the task objective, business context, judgment criteria, and boundary conditions
Schema	Define the output structure, field types, enum values, and required fields
Validator	Execute deterministic rules, such as amount ranges, date validity, and field combination relationships
Evaluator	Offline evaluation of content quality, such as factual consistency, recall, and human agreement rate

The schema should not become a “second prompt.” Field descriptions can help the model understand, but they cannot replace business rules.

4. Complex Tasks Should Be Split First, Then Structured

If a single schema requires the model to perform classification, extraction, explanation, scoring, evidence citation, and suggestion generation all at once, the failure rate and debugging difficulty will both increase. In production, it is better to split the task into multiple stages:

• Step 1: Document classification → small schema
• Step 2: Field extraction → extraction schema
• Step 3: Risk assessment → assessment schema
• Step 4: Result explanation → readable text or explanation schema

The advantage of this approach is that each step can be evaluated independently, and it is easier to set different models, different temperatures, and different retry strategies for each.

Applicable Scenarios

Scenario	Description
Information Extraction	Contract elements, invoice fields, medical record summaries, ticket labels, customer service intent recognition. Structured outputs can reduce backend parsing glue code, especially for tasks with stable fields and clear validation rules
Tool Call Parameter Generation	When an agent calls a tool, the parameters must be parseable, verifiable, and auditable. Structured Outputs can reduce parameter format errors, but permission checks and business confirmation are still needed before tool execution
Frontend UI Data Generation	Generating form configurations, chart data, recommendation cards, step lists. A schema can constrain the fields required for frontend rendering, preventing UI crashes due to missing fields
Batch Processing and Offline Data Cleaning	For large-scale data extraction, structured outputs can reduce manual cleaning costs. However, you must monitor parse error rates, retry rates, field null rates, and human spot-check agreement rates

Common Misconceptions

Misconception 1: Using Structured Outputs Means No Retries

Retries are still necessary, but they must have boundaries. It is recommended to distinguish:

Failure Type	Strategy
Parse Failure	Short retry possible
Schema Incompatibility	Do not blindly retry; alert instead
Business Rule Failure	Usually should not be retried; should enter a manual or degraded path
Model Refusal	Handle according to safety policy, not by force-fixing

Misconception 2: The More Detailed the Schema, the More Stable

Overly deep, overly wide, or overly long enum schemas increase the burden on the model and the decoding backend. Fields should be as close as possible to the actual downstream needs. Fields that can be calculated through business rules should not be generated by the model.

Misconception 3: Only Look at Parse Success Rate

The parse success rate can easily give a team a false sense of security. What matters more are: field accuracy rate, enum confusion rate, null rate, human agreement rate, downstream rollback rate, and user correction rate.

Misconception 4: Execute JSON Results Directly

Especially in tool calling and agent scenarios, structured parameters are not equivalent to authorization. The JSON generated by the model is only a candidate decision. For side-effect operations involving payments, deletions, sending messages, or modifying permissions, you must go through permission boundaries and confirmation processes.

Go-Live Checklist

Schema Design

• Each schema has a unique name and version number
• Required fields come from real downstream dependencies, not just “to be complete”
• Enum values are stable, short, and have clear meanings
• Semantics for null, unknown, and empty are defined
• Schema changes have compatibility checks

Inference and Parsing

• Use structured output capabilities that support JSON Schema / grammar / regex
• Post-generation JSON parser and schema validator are not removed
• Business rule validation is independent of model output
• Unified error codes for refusal, truncation, timeout, and empty output
• Retry count, retry conditions, and backoff strategy are configured

Quality Evaluation

• Have a golden set covering normal, edge, dirty data, and adversarial examples
• Evaluate field-level accuracy, not just overall success rate
• Human spot-checks and model evaluations are recorded separately
• Perform regression comparisons across different models and schema versions

Performance and Cost

• Monitor TTFT, output latency, P95/P99, tokens per second
• Monitor constrained decoding overhead
• Implement compression, splitting, or fallback strategies for complex schemas
• Set independent concurrency and cost budgets for batch tasks

Safety and Governance

• Tool call parameters undergo permission checks before execution
• Output content is checked for sensitive information and injection before entering downstream systems
• Schemas and prompts are under version control
• Error samples are traceable to input, model, schema, prompt, and version

A Simplified Call Structure

The example below shows the structure that the application layer should care about, without being tied to a specific vendor’s SDK:

from pydantic import BaseModel, Field
from typing import Literal, List

class TriageResult(BaseModel):
    intent: Literal["claim", "renewal", "complaint", "unknown"]
    confidence: float = Field(ge=0, le=1)
    missing_fields: List[str]
    should_handoff: bool
    reason: str

def run_structured_generation(client, model: str, user_text: str) -> TriageResult:
    schema = TriageResult.model_json_schema()
    response = client.generate(
        model=model,
        input=user_text,
        response_schema=schema,
        strict=True,
    )
    parsed = TriageResult.model_validate_json(response.text)
    if parsed.confidence < 0.6:
        parsed.should_handoff = True
    return parsed

This code expresses an engineering principle: the schema is generated by the type system, the model output enters type validation, and business rules are executed in post-processing logic.

References

1. OpenAI, Structured model outputs. https://developers.openai.com/api/docs/guides/structured-outputs
2. vLLM Documentation, Structured Outputs. https://docs.vllm.ai/en/latest/features/structured_outputs/
3. LMSYS, Fast JSON Decoding for Local LLMs with Compressed Finite State Machine. https://www.lmsys.org/blog/2024-02-05-compressed-fsm/
4. dottxt-ai, Outlines: Structured Outputs. https://github.com/dottxt-ai/outlines
5. Geng et al., JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models. https://arxiv.org/abs/2501.10868
6. Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
7. Le, Schema Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding. https://arxiv.org/abs/2604.14862