Is LLM Batch Inference suitable for real-time chat?

No. It is designed for offline tasks that can tolerate delays. Real-time chat, voice interactions, and low-latency APIs should still use online inference pipelines.

Are Batch Inference and Continuous Batching the same thing?

No. Batch Inference is an offline asynchronous task pattern, while Continuous Batching is a dynamic scheduling technique within online inference services.

Should I retry the entire batch if a batch fails?

Not recommended. A safer approach is to merge results using custom_id or business task IDs, and only retry failed, expired, or recoverable sub-tasks.

Can Batch Inference replace online APIs?

No. It is designed for offline tasks that can tolerate delays ranging from minutes to hours, and is not suitable for scenarios requiring immediate responses.

LLM Batch Inference in Production: Running Offline LLM Tasks with Async Batch Processing

Background

Many teams, when first integrating LLMs, tend to funnel all tasks through online APIs: user requests go through the online interface, background summarization goes through the online interface, log classification goes through the online interface, and data labeling also goes through the online interface. While this is simple in the short term, it leads to three problems over time.

First, offline tasks compete with real-time traffic. Tasks like batch summarization, historical ticket categorization, knowledge base entry cleaning, and evaluation set replay typically don’t need sub-second responses, yet they compete with real user requests for the same online rate limits and concurrency resources.

Second, failure recovery is coarse. If 800 out of 100,000 tasks fail, many systems simply retry the entire batch, resulting in duplicate costs, duplicate writes, and difficult result reconciliation.

Third, cost and throughput are uncontrollable. Offline tasks are naturally suited for queuing, peak shaving, sharding, and low-priority processing. If handled as online tasks, teams struggle to manage cost, throughput, and completion time windows separately.

The core value of LLM Batch Inference is transforming LLM calls that “don’t need immediate responses” into queueable, traceable, and recoverable asynchronous jobs. The OpenAI Batch API is explicitly designed for asynchronous request groups, offering a dedicated batch processing interface, a higher independent rate limit pool, a 24-hour completion window, and cost savings. Anthropic Message Batches also targets large-scale asynchronous Messages requests, ideal for high-volume tasks that don’t require instant responses. Similarly, Google Gemini Batch Inference and Amazon Bedrock Batch Inference treat batch prompts as asynchronous jobs.

Core Principles

1. Batch Inference is Not Online Batch Processing

First, let’s distinguish two easily confused concepts.

Concept	Continuous Batching	Batch Inference
Role	Scheduling technique within online inference services	Offline asynchronous task pattern
Goal	Increase online throughput while controlling latency	Improve throughput and reduce cost within an acceptable time window
Mechanism	Dynamically combines multiple in-flight requests by token iteration	Prepare input → Submit job → Poll/callback → Download results
Response Time	Milliseconds to seconds	Minutes to hours
Use Cases	Online chat, real-time conversations	Offline summarization, batch classification, data labeling, evaluation replay

This means the design focus of Batch Inference is not “how to make a single user see a response faster,” but “how to stably, recoverably, and accountably process 100,000 or 1,000,000 offline tasks.”

2. Job Input is Typically File-Based Records

Batch processing interfaces on major platforms usually require organizing each request as an individual record. For example, the OpenAI Batch API uses a JSONL file where each line is a request, and custom_id links input and output. Anthropic Message Batches also recommends handling results based on custom_id and result type. Bedrock Batch Inference requires uploading input files to S3, submitting the batch job, and then retrieving the output from S3.

A production-ready input record should contain at least three types of IDs:

{
  "custom_id": "ticket-summary:2026-07-02:000001",
  "method": "POST",
  "url": "/v1/responses",
  "body": {
    "model": "gpt-4.1-mini",
    "input": "Summarize this support ticket...",
    "metadata": {
      "job_id": "job_20260702_ticket_summary",
      "source_id": "ticket_000001",
      "pipeline_version": "summary_v3"
    }
  }
}

custom_id should not be just a random UUID. A better practice is to encode the business object, task type, date, or batch number into it, facilitating failure recovery, duplicate submission detection, and result merging.

3. Output is Not Reliably Ordered by Input

Batch processing results are typically a single output file, with each line corresponding to a success or failure result. In engineering, you cannot assume the output order matches the input order, nor should you rely solely on array indices for merging. A safer approach is to always use custom_id / source_id / job_id for result correlation.

A results table can be designed as follows:

CREATE TABLE llm_batch_result (
    id          BIGSERIAL PRIMARY KEY,
    job_id      TEXT NOT NULL,
    custom_id   TEXT NOT NULL,
    source_id   TEXT NOT NULL,
    status      TEXT NOT NULL,
    model_name  TEXT,
    input_tokens   INTEGER DEFAULT 0,
    output_tokens  INTEGER DEFAULT 0,
    result_json    JSONB,
    error_code     TEXT,
    error_message  TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(job_id, custom_id)
);

UNIQUE(job_id, custom_id) is key. It prevents duplicate imports, duplicate compensations, and duplicate billing statistics.

Engineering Implementation

1. First, Determine if the Task is Suitable for Batch Processing

Tasks suitable for Batch Inference typically have three characteristics:

Users do not wait for results; minute-level or hour-level completion is acceptable.
The input set is well-defined and can be split into many independent records.
Results can be asynchronously backfilled, e.g., written to a database, object storage, search index, or evaluation report.

Typical Scenarios:

Scenario	Description
Historical customer service ticket summarization	Batch generate summaries, sentiment, issue categories
Comment/feedback/email classification	Label large volumes of text
Document metadata extraction	Batch extract structured fields
Data labeling and weak label generation	Automatically generate training data labels
RAG knowledge base entry cleaning	Batch clean and standardize knowledge base entries
Offline evaluation replay	Batch evaluation after model or prompt upgrades
Low-priority content generation	Batch generate non-real-time content

Unsuitable Scenarios: Real-time chat, real-time customer service agent assistance, voice agents, form submissions where users are waiting, and tool-calling chains requiring multi-turn interactions. Bedrock documentation explicitly states that its batch processing does not support tool calling/function calling or structured output, which require client-side back-and-forth interactions.

2. Split Large Tasks into Recoverable Shards

Don’t concatenate all inputs into one giant batch. A more robust structure is:

batch_job
├── shard_0001.jsonl
├── shard_0002.jsonl
├── shard_0003.jsonl
└── shard_0004.jsonl

Each shard has an independent state: created → uploaded → submitted → running → completed / failed / expired → imported. The advantage is that if one shard fails, you only need to retry that shard, not the entire task.

Shard size should be determined based on platform limits, task complexity, and downstream import capacity. For example, OpenAI documentation states that a single batch can contain up to 50,000 requests, with a maximum input file size of 200 MB; Gemini Batch Inference documentation states that a single batch job can contain up to 200,000 requests. In practice, you don’t always need to hit the upper limit. It’s safer to start with smaller shards to validate failure rates, processing times, and import speeds.

3. Establish Idempotent Submission and Result Import

The batch processing pipeline requires at least two layers of idempotency.

The first layer is submission idempotency. The same job_id + shard_id should not be submitted as multiple platform jobs unless the old job has explicitly failed, been canceled, or expired.

The second layer is result import idempotency. When the same job_id + custom_id is imported multiple times, it should only update the same record, not insert duplicate results.

def import_batch_results(job_id: str, output_lines: list[dict]) -> None:
    for line in output_lines:
        custom_id = line["custom_id"]
        result = normalize_result(line)
        upsert_result(
            job_id=job_id,
            custom_id=custom_id,
            source_id=parse_source_id(custom_id),
            status=result.status,
            model_name=result.model,
            input_tokens=result.input_tokens,
            output_tokens=result.output_tokens,
            result_json=result.body,
            error_code=result.error_code,
            error_message=result.error_message,
        )

4. Don’t Treat All Failures the Same

Batch processing failures can generally be categorized into four types:

Failure Type	Description	Handling Strategy
Input validation failure	Invalid request format, model name, or fields	Fix the generation logic and retry; do not blindly retry
Recoverable service error	Temporary service outage, rate limiting, internal error	Retry based on failed records
Expired/unfinished	Batch window expired, some requests not executed	Retry only expired records
Business-unprocessable	Empty input, text too long, sensitive content not allowed	Fall back to manual or rule-based handling

The OpenAI Batch API writes failed request errors to an error file, and you can use custom_id to locate expired requests. Anthropic’s examples also show handling results based on succeeded, errored, and expired result types. Production systems should map these states to their own unified error model.

5. Establish a Batch Task Ledger

It’s recommended to create three separate tables: llm_batch_job, llm_batch_shard, and llm_batch_result.

CREATE TABLE llm_batch_job (
    job_id          TEXT PRIMARY KEY,
    task_type       TEXT NOT NULL,
    model_name      TEXT NOT NULL,
    prompt_version  TEXT NOT NULL,
    status          TEXT NOT NULL,
    total_items     INTEGER NOT NULL DEFAULT 0,
    succeeded_items INTEGER NOT NULL DEFAULT 0,
    failed_items    INTEGER NOT NULL DEFAULT 0,
    expired_items   INTEGER NOT NULL DEFAULT 0,
    created_by      TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    finished_at     TIMESTAMPTZ
);

CREATE TABLE llm_batch_shard (
    shard_id         TEXT PRIMARY KEY,
    job_id           TEXT NOT NULL REFERENCES llm_batch_job(job_id),
    input_uri        TEXT NOT NULL,
    provider_batch_id TEXT,
    output_uri       TEXT,
    error_uri        TEXT,
    status           TEXT NOT NULL,
    retry_count      INTEGER NOT NULL DEFAULT 0,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

The ledger isn’t for show; it’s to answer several production questions:

Has this batch of tasks been fully completed?
Which shards failed, and what were the failure reasons?
Which source_ids have no results?
Will a retry cause duplicate writes?
What are the cost and failure rate for a specific prompt_version?

Use Cases

Offline Summarization

Generate summaries, sentiment, issue categories, and suggested actions for customer service tickets from the past year. This type of task is high-volume, but users don’t need real-time results, making it ideal for batch processing.

Batch Classification

Classify comments, emails, customer service conversations, and public opinion texts with labels. The key is ensuring that input versions, prompt versions, model versions, and output results are aligned; otherwise, it becomes difficult to explain why classification criteria changed later.

Large-Scale Evaluation Replay

When prompts or models are upgraded, you can package historical samples, ground truth answers, and scoring rules into a batch processing task to run an offline evaluation at low cost. Note that this should not be confused with LLM-as-a-Judge: this article focuses on batch processing task orchestration, not the automatic scoring method itself.

Data Labeling and Cleaning

Batch processing can be used to generate weak labels, extract structured fields, clean long texts, and supplement metadata. However, such tasks must retain manual spot-checking and version records, because errors generated in batch can be systematically amplified.

Common Misconceptions

Misconception 1: Treating Batch Inference as a Cheaper Online API

Batch processing is not a low-latency interface. It is suitable for tasks that can tolerate delays. If users are waiting for a response, batch processing will only worsen the experience.

Misconception 2: Only Recording the Platform batch_id, Not Business IDs

The platform batch_id only identifies the job on the platform side; it doesn’t directly explain the business object. Production systems must save job_id, shard_id, custom_id, source_id, prompt_version, and model_name.

Misconception 3: Retrying the Entire Batch on Failure

Retrying the entire batch increases costs and can lead to duplicate writes. The correct approach is to split the retry set based on result status, only handling failed, expired, or missing records.

Misconception 4: Ignoring Output Order Issues

Output files should not be forcibly mapped one-to-one with input files by line number. Result merging must be based on custom_id.

Misconception 5: Not Estimating Costs

While batch processing may be cheaper, it doesn’t mean there is no cost risk. Before submission, estimate input tokens, maximum output tokens, the number of shards, and the expected failure retry ratio. For large-volume jobs, set budget thresholds and approval gates.

Pre-Launch Checklist

Before going live, at least check the following:

Recommended Production Architecture

You can build the following pipeline:

Business Data Source → Task Filtering & Snapshot → Prompt/Model Version Binding
    → JSONL Shard Generation → Object Storage Upload → Batch Job Submission
    → Status Polling or Event Notification → Output File Download
    → Result Merging & Idempotent Write → Failed Record Retry → Sampling QC & Reports

The key to this pipeline is the “snapshot”. Batch processing tasks often run for a long time. If business data is modified during task execution, you must be able to explain which version of input, which version of prompt, and which version of model generated the output.