LLM Batch API Offline Inference Production Guide: Reducing Costs with JSONL Queues, custom_id, and Result Backfill

Background: Not All LLM Requests Should Go Through Online APIs

Many teams, after integrating LLMs, funnel all tasks into the same online call chain: user requests, background classification, batch summarization, historical data labeling, knowledge base embeddings, evaluation set replays, and risk control content audits—all via synchronous APIs. Initially, with low request volumes, this approach is simple and straightforward; but as data scales up, problems become apparent.

The first issue is cost and rate limiting. Background tasks typically don’t need second-level responses, yet they compete with online user requests for synchronous API quotas. If an overnight classification task exhausts the rate limit, daytime user request latency and failure rates will be dragged down by unrelated tasks.

The second issue is recoverability. When a synchronous script fails midway, the common result is that no one knows which samples have been processed, which need retries, and which outputs have been written back to the business database. Rerunning the whole thing might seem simple, but it can actually cause duplicate billing, duplicate writes, and result drift.

The third issue is result reconciliation. Offline tasks usually involve tens of thousands or millions of samples. What’s needed is a job-level perspective: input file version, task batch, completion rate, failure reasons, token usage, and result storage status—not temporary logs of individual synchronous requests.

Therefore, the value of Batch API is not just “a bit cheaper.” More precisely, it transforms non-real-time LLM requests from an online call pattern into an offline inference pipeline that is queueable, traceable, recoverable, and reconcilable.

Core Principles of Batch API

The Batch API designs of major model platforms are largely similar: developers prepare a batch of requests, usually as a JSONL file or request array; the platform processes them asynchronously; developers poll the job status; after completion, they download the result file or read the result stream; finally, they write the results back to internal systems using custom IDs.

Platform	Input Format	ID Field	Key States	Features
OpenAI	JSONL file	custom_id	validating → in_progress → finalizing → completed	Supports evaluation, classification, embedding, etc.
Anthropic	Messages request array	custom_id	in_progress → ended	Explicitly lists succeeded/errored/canceled/expired
Gemini	Inline requests or JSONL	key	pending → running → succeeded/failed/cancelled/expired	Supports batch job cancellation

While the details differ, the engineering abstraction can be unified into five core objects:

• Input Manifest: A frozen input inventory containing business primary keys, input hashes, prompt versions, and model parameters.
• Batch Job: An asynchronous job submitted to the model platform.
• Status Poller: A status poller that maps platform states to an internal standard state machine.
• Result Sink: A result landing layer that first stores raw responses before performing business backfill.
• Retry Ledger: A retry ledger that categorizes failures by error type for processing.

Task Types Suitable for Batch API

Batch API is best suited for tasks where “throughput is more important than immediacy.”

Typical Scenarios:

• Offline Evaluation: Batch-run model outputs on golden sets, regression sets, and red-teaming samples, archiving by version.
• Large-Scale Classification: Batch-label tickets, comments, products, and knowledge base fragments.
• Batch Summarization: Generate summaries or structured fields for historical customer service conversations, meeting minutes, and long documents.
• Embedding Construction: Perform initial embeddings or periodic recalculations on content libraries.
• Data Cleaning: Use models to assist in identifying dirty data, duplicates, anomalous fields, or compliance risks.
• Non-Real-Time Content Generation: Batch-generate product descriptions, SEO drafts, and internal knowledge Q&A drafts.

Scenarios that are not suitable are also clear: online chat, real-time voice, user interactions with streaming input/output, Agent workflows requiring real-time tool call confirmation, and strongly real-time pipelines like payment risk control.

Production Architecture: Treat Batch as a Job System, Not a Script

A maintainable Batch API pipeline typically consists of seven layers.

1. Task Entry: Convert Business Needs into batch_task

Don’t let business scripts directly generate JSONL and call the model platform. A safer approach is to first create an internal batch_task record, documenting the task type, model, prompt version, input data version, estimated request count, budget cap, initiator, approval status, and expiration policy.

CREATE TABLE llm_batch_task (
  id TEXT PRIMARY KEY,
  task_type TEXT NOT NULL,
  model_name TEXT NOT NULL,
  prompt_version TEXT NOT NULL,
  input_manifest_uri TEXT NOT NULL,
  status TEXT NOT NULL,
  total_items INTEGER NOT NULL,
  estimated_input_tokens BIGINT,
  estimated_output_tokens BIGINT,
  budget_cents BIGINT,
  provider_batch_id TEXT,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL
);

The key point here is: Batch jobs must first become auditable objects within your internal system before becoming asynchronous tasks on external model platforms.

2. Input Manifest: Freeze the Input Version with a Manifest

The biggest fear for Batch tasks is “uncertain input.” If you query the database in real-time for pending data, a retry might fetch a different set of data, making results irreproducible.

A better approach is to generate an Input Manifest: freeze the input records for this task into an inventory containing business primary keys, input hashes, prompt versions, model parameters, output schema versions, and custom IDs.

Don’t just use random UUIDs for custom_id. A recommended structure is:

{task_id}:{item_type}:{business_id}:{input_hash_prefix}

For example: task_20260706_ticket_label:ticket:TK2026003912:9f3a21bc

This has three benefits:

1. You can trace back to business records from the results.
2. You can identify if the same business record has been reprocessed due to input changes.
3. You can perform idempotency checks during backfill.

3. JSONL Construction: Validate Line by Line, Don’t Fail Entirely

The advantage of JSONL is its simplicity, streamability, and support for sharded uploads. The risk is that a single malformed line can cause the entire batch validation to fail.

When generating JSONL, perform three types of validation locally:

• Structural Validation: Each line must be valid JSON and contain the fields required by the platform.
• Budget Validation: Estimate input tokens and maximum output tokens to prevent individual requests from ballooning abnormally.
• Policy Validation: Prohibit writing sensitive fields, unsanitized original text, or data that shouldn’t leave the domain into the batch file.

A simplified JSONL line example:

{"custom_id":"task_20260706_ticket_label:ticket:TK2026003912:9f3a21bc","method":"POST","url":"/v1/responses","body":{"model":"gpt-5.5-mini","input":"Please classify the following ticket as billing, technical, or account.\\n\\nTicket content: ...","max_output_tokens":200}}

4. Job Submission: Sharding, Naming, and Quota Protection

In production, don’t cram all requests into a single giant batch. A safer approach is to shard by task type, model, input size, and business priority. For example, a million-level classification task can be split into 100 shards, each submitted, polled, and backfilled independently.

Benefits of sharding:

Advantage	Description
Fault Isolation	Failure of one shard doesn’t affect others.
Concurrency Control	You can control the number of concurrent submissions.
Cost Statistics	You can track failure rates and costs per shard.
Priority Scheduling	High-priority tasks can be inserted without waiting for a giant job to finish.

Before submission, check the external platform’s batch limits (max requests per batch, file size, queued token cap, creation frequency, job validity period, etc.). Put these limits in the provider adapter configuration, not hardcoded in business logic.

5. Status Polling: The State Machine Must Be Explicit

Batch jobs are inherently asynchronous, making the state machine more important than a one-shot script. An internal state can be designed as:

CREATED → VALIDATING → SUBMITTED → RUNNING → FINALIZING → COMPLETED
                                                      → FAILED
                                                      → EXPIRED
                                                      → CANCELLED

The poller must do three things:

1. Periodically query the provider’s batch status and map the raw status to an internal standard state.
2. Record status change timestamps to calculate queue time, run time, and total completion time.
3. Handle stuck jobs—if a task remains in the submitted state for too long, it should trigger manual review or automatic split-and-retry, not infinite polling.

6. Result Backfill: Reconcile by custom_id, Not by Line Number

Result backfill is the most error-prone part of Batch API. The output file is usually also JSONL, but the output order may not match the input order. The correct approach is to read each line, parse the custom_id or key, look up the internal manifest, and confirm that the input hash, task ID, business ID, and output schema version all match before writing back.

It’s recommended to split the results table into two layers:

CREATE TABLE llm_batch_result_raw (
  id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  custom_id TEXT NOT NULL,
  provider_request_id TEXT,
  result_type TEXT NOT NULL,
  raw_json TEXT NOT NULL,
  input_tokens INTEGER,
  output_tokens INTEGER,
  created_at TIMESTAMP NOT NULL
);

CREATE TABLE llm_batch_result_applied (
  id TEXT PRIMARY KEY,
  task_id TEXT NOT NULL,
  business_id TEXT NOT NULL,
  custom_id TEXT NOT NULL,
  output_version TEXT NOT NULL,
  applied_status TEXT NOT NULL,
  applied_at TIMESTAMP
);

The first table retains the provider’s raw results for auditing and re-parsing. The second table records whether the result has been written back to the business system, preventing duplicate writes.

7. Retry Ledger: Failure is Not a Single State, But a Set of Reasons

Failed results should be categorized into at least five types:

Error Type	Description	Handling Strategy
validation_error	Input structure or parameter error	Correct the request and resubmit.
rate_or_capacity_error	Platform capacity or queuing limit	Split or retry later.
expired	Job not completed within the window	Reduce shard size or adjust submission cadence.
content_policy_error	Content blocked by safety policy	Route to compliance processing.
internal_error	Platform or network anomaly	Retry with idempotency strategy.

Don’t simply rerun all failures. Rerunning validation_error will fail again; blindly rerunning content_policy_error may expand compliance risks; directly rerunning expired may continue to expire. The retry system should take different actions based on the error type.

Common Misconceptions

Misconception 1: Equating Batch API with Online Dynamic Batching

Batch API is an asynchronous job interface provided by the platform for non-real-time tasks. Continuous batching or in-flight batching is a scheduling strategy within the inference service for online throughput optimization. Both are called batching, but their engineering boundaries are completely different. This article discusses the former: how to organize a large number of non-real-time requests into traceable offline jobs.

Misconception 2: Only Looking at Discounts, Ignoring Capital Tie-Up and Expiration Risk

Batch API often has cost advantages, but that doesn’t mean you can submit indefinitely. Batch tasks can consume budgets, hit queuing limits, or even expire if too large. Production systems should estimate tokens and budgets before submission, monitor completion rates during runtime, and retry only incomplete samples after expiration.

Misconception 3: Using Auto-Incrementing Line Numbers for Result Mapping

Line numbers may seem stable in the input file, but they are not reliable in the output file. Results should be mapped using custom_id, key, or internal business primary keys. Otherwise, if the output order changes, results for user A could be written to user B’s record.

Misconception 4: Hardcoding Prompts in Scripts

Offline batch processing is often run repeatedly. If prompts are hardcoded in scripts, it will be difficult to explain why a certain batch of results was generated a certain way six months later. Prompts, models, temperatures, max outputs, and output schemas should all be versioned and written into the task and manifest.

Misconception 5: Only Saving Final Results, Not Raw Responses

Saving only business fields loses troubleshooting information. The raw response’s request ID, usage, finish reason, error type, and model version can all be valuable for auditing and cost analysis. It’s recommended to land raw data first, then parse and perform business backfill.

Recommended Offline Inference Pipeline

The complete flow is as follows:

Business System → Create batch_task → Generate input_manifest → Build JSONL shard
→ Upload and submit provider batch → Poll batch status → Download output/error JSONL
→ Write to raw_result → Parse structured results → Idempotent backfill to business tables
→ Generate cost, failure rate, and quality reports

Four key control points:

1. Pre-Submission Gate: Check input size, sensitive fields, prompt version, model availability, budget cap, output schema, and dry-run samples.
2. Runtime Observability: Monitor the number of submitted, running, completed, expired, and failed shards; track queue time, completion rate, and estimated remaining time.
3. Post-Completion Reconciliation: The number of items in the input manifest should match the sum of succeeded, failed, expired, and missing items, as well as the number of business backfills.
4. Retry and Review: Only retry retryable items, and write the error reason, retry count, and final status to the ledger.

Code Example: Idempotent Backfill Using custom_id

The following example demonstrates the core idea: read the provider’s output JSONL, write results to raw storage by custom_id, and then hand them off to the business backfill processor.

import fs from "node:fs";
import readline from "node:readline";

type BatchLine = {
  custom_id?: string;
  key?: string;
  response?: unknown;
  result?: unknown;
  error?: unknown;
};

type ManifestItem = {
  taskId: string;
  customId: string;
  businessId: string;
  inputHash: string;
};

async function loadManifestByCustomId(customId: string): Promise<ManifestItem | null> {
  // In production, query the database or object storage for the manifest index.
  return null;
}

async function saveRawResult(args: {
  taskId: string;
  customId: string;
  businessId: string;
  resultType: "succeeded" | "errored" | "expired" | "unknown";
  rawJson: string;
}) {
  // First write to raw_result, with a unique index on taskId + customId to ensure idempotency.
}

function classifyResult(line: BatchLine): "succeeded" | "errored" | "expired" | "unknown" {
  if (line.error) return "errored";
  const resultType = (line.result as any)?.type;
  if (resultType === "succeeded") return "succeeded";
  if (resultType === "errored") return "errored";
  if (resultType === "expired") return "expired";
  if (line.response) return "succeeded";
  return "unknown";
}

export async function importBatchOutput(filePath: string) {
  const stream = fs.createReadStream(filePath, { encoding: "utf8" });
  const rl = readline.createInterface({ input: stream, crlfDelay: Infinity });

  for await (const rawLine of rl) {
    if (!rawLine.trim()) continue;
    const line = JSON.parse(rawLine) as BatchLine;
    const customId = line.custom_id ?? line.key;
    if (!customId) {
      throw new Error(`missing custom_id/key in output line: ${rawLine.slice(0, 200)}`);
    }
    const manifest = await loadManifestByCustomId(customId);
    if (!manifest) {
      throw new Error(`orphan batch result: ${customId}`);
    }
    await saveRawResult({
      taskId: manifest.taskId,
      customId,
      businessId: manifest.businessId,
      resultType: classifyResult(line),
      rawJson: rawLine,
    });
  }
}

This code deliberately does not write results directly into business tables. In production, raw storage should be completed first, then the parser performs structural validation against the output schema, and finally the backfill processor idempotently writes to business fields.

Go-Live Checklist

Before going live, check each item:

1. Task Modeling: Do you have batch_task, manifest, raw_result, applied_result, and retry_ledger?
2. Input Freezing: Can you reproduce the complete input, prompt, model, and parameters for a given batch?
3. ID Design: Can custom_id trace back to business records? Does it comply with each platform’s character and length limits?
4. Local Validation: Is JSONL validated line by line? Are tokens and budget estimated?
5. Status Polling: Are states like running, completed, failed, expired, and cancelled handled?
6. Result Mapping: Is reconciliation done only by custom_id/key, with zero reliance on output order?
7. Error Classification: Are validation errors, platform errors, expiration, cancellation, and content policy errors distinguished?
8. Idempotent Backfill: Will re-importing the same result file not cause duplicate writes to business tables?
9. Budget Protection: Are there budget caps at the task, tenant, and model levels?
10. Data Retention: Do you understand the platform’s retention policy for Batch input/output? Is sensitive data handled according to business compliance requirements?
11. Manual Review: Are high-risk tasks sampled for review? Is there a rollback or revocation marking mechanism?
12. Reporting: Can you output success rate, failure rate, expiration rate, token consumption, unit cost, and business backfill completion rate?

Summary

Batch API is not just a “cheaper online API”—it’s a completely different engineering paradigm. Migrating non-real-time LLM tasks from synchronous call patterns to an offline inference pipeline requires establishing a complete job lifecycle management system: from task modeling, input freezing, JSONL sharding, and status polling, to custom_id result reconciliation, error classification retries, and idempotent backfill. When your system can answer, from a job perspective, “how many tasks were processed, which succeeded, which failed, and how much it cost,” you have truly mastered the production practice of Batch API.

References

• OpenAI Batch API
• Anthropic Claude Message Batches
• Google Gemini Batch API
• Google Cloud Gemini Batch Inference