Does a batch job status of 'success' mean every record succeeded?

No. A job-level terminal status only indicates the scheduling flow has ended. You must still reconcile each record for success, failure, missing, and duplicates against the expected set from the input manifest.

Can I resubmit a batch job immediately after a submission timeout?

No. You should first check your local submission ledger, input file hash, and the provider's job list to confirm whether the job was already created. Otherwise, you risk creating a second job and incurring duplicate charges.

Should a failed retry resubmit the entire batch?

Generally no. Extract only retryable failures and missing records from the reconciliation results, generate a new retry shard, and preserve the parent record ID and retry attempt number.

Why can't I use the input line number directly as the Record ID?

Sharding, merging, filtering, and retries all change line numbers. A stable ID should come from the business primary key, source data version, prompt version, and model configuration to ensure cross-file and cross-attempt correlation.

LLM Batch Inference Production Practices: Avoiding Duplicate Charges and Missing Results with Manifest, Idempotent Record IDs, and Shard Retry

Background: Batch APIs Are Cheap, But Not as Simple as “Upload a File and Wait for Results”

Large model batch inference is ideal for offline extraction, data labeling, evaluation, content classification, embedding generation, and historical data backfill. It typically submits JSONL files or inline requests, completes asynchronously within hours, and often costs less than real-time interfaces.

However, once you go into production, the risks aren’t in the model call itself, but in the often-overlooked corners of the job lifecycle:

The client request times out, but the server has already created the job. A retry generates a second charge.
The output file order differs from the input order. Backfilling by line number writes results to the wrong business records.
The job shows as completed, but some records failed, expired, or have no output at all.
After canceling a job, partial results still exist, but the system treats the entire batch as failed.
Rerunning the entire input to fix a few failed records duplicates charges and overwrites already-validated results.
Provider result files have retention limits. If you don’t download and archive them in time, you can’t re-reconcile later.

Therefore, the core of production-grade batch inference isn’t “polling for status,” but establishing a provable one-time submission, a per-record traceable result set, and a compensation loop that retries only the necessary records.

Core Principle: Decompose Batch Inference into Two Layers of Idempotency

Job-Level Idempotency

The job layer answers one question: Has this input already been successfully submitted?

Don’t rely solely on the provider’s returned Batch ID, because the Batch ID only exists after the creation request succeeds. A safer approach is to generate a local job_key before submission:

job_key = sha256(
    tenant_id + dataset_version + input_manifest_hash +
    model_id + prompt_version + generation_config_hash
)

The job_key corresponds to a local job record. The state machine should include at least:

State	Meaning
`CREATED`	Job created, awaiting validation
`VALIDATED`	Input validation passed
`SUBMITTING`	Submitting to provider
`SUBMITTED`	Provider confirmed receipt
`RUNNING`	Provider processing
`RECONCILING`	Reconciling results
`PARTIAL`	Some records need compensation
`COMPLETED`	All reconciliation passed
`FAILED`	Job-level failure
`CANCELLED`	Cancelled
`EXPIRED`	Results expired

The program must first persist the SUBMITTING state and the input file hash, then call the provider’s creation API. If a network timeout occurs, the recovery program should first check the local ledger and the remote job, not directly resubmit.

Note: Google Gemini’s current documentation explicitly states that batch job creation is not idempotent; submitting the same creation request twice generates two independent jobs. This means “HTTP retry” cannot be directly equated to “business retry.”

Record-Level Idempotency

The record layer answers another question: Has each business input received a unique, verifiable result?

The record ID should come from a stable business field, not a temporary line number:

record_id = sha256(
    tenant_id + business_primary_key + source_version +
    prompt_version + model_config_version
)

This way, the same business record with the same input and configuration gets the same ID; when the prompt, model parameters, or source data change, the ID changes accordingly.

OpenAI Batch uses a unique custom_id to map output back to input; Amazon Bedrock uses recordId, and the official documentation explicitly states that the output JSONL record order is not guaranteed to match the input. Regardless of whether the provider promises to maintain order, production systems should always join by record ID, not by line number.

Manifest: The Source of Truth for Batch Jobs

Each submission should generate an immutable Manifest. It serves as both an input snapshot and the basis for subsequent reconciliation, auditing, and retries.

{
  "job_key": "sha256:...",
  "manifest_version": 1,
  "tenant_id": "tenant-a",
  "provider": "openai",
  "model": "model-version",
  "endpoint": "/v1/responses",
  "prompt_version": "summary-v7",
  "generation_config_hash": "sha256:...",
  "input_file_sha256": "sha256:...",
  "expected_record_count": 50000,
  "shards": [
    {
      "shard_id": "part-00017",
      "file_sha256": "sha256:...",
      "record_count": 2000
    }
  ],
  "created_at": "2026-07-28T02:59:08-04:00"
}

The Manifest must freeze at least the following information:

Input record set and its hash.
Exact model version, not a floating alias.
Prompt, System Instruction, Tool Schema, and generation parameter versions.
Shard IDs, record counts, byte counts, and file hashes.
Submitter, tenant, cost center, and data retention policy.
Expected output schema and its version.

Input files, Manifests, and result files should be saved with append-only or content-addressed storage, not overwritten in place.

Submission Flow: Log First, Then Call the Provider

Use an Outbox pattern or database unique constraints to protect the submission process:

def submit_batch(job_key: str) -> str:
    job = load_job_for_update(job_key)
    if job.provider_job_id:
        return job.provider_job_id
    assert job.input_validated
    mark_submitting(job_key)

    remote = provider.find_by_metadata(job_key)
    if remote:
        bind_provider_job(job_key, remote.id)
        return remote.id

    created = provider.create_batch(
        input_file_id=job.input_file_id,
        metadata={"job_key": job_key},
    )
    bind_provider_job(job_key, created.id)
    return created.id

The key constraints aren’t this code itself, but the following guarantees:

job_key must be unique in the local database.
Only one submitter can hold a lease for the same job.
JSONL schema, record ID uniqueness, and file hash validation are completed before submission.
After a request timeout, check the remote first; do not immediately resend.
When the provider supports metadata, write the job_key to the remote job for reverse lookup.

Result Reconciliation: Terminal Status Isn’t Completion; Set Closure Is

Let the expected record set from the Manifest be E, successful results be S, and terminal failures be F:

missing   = E − (S ∪ F)
unknown   = (S ∪ F) − E
duplicate = record_ids with count > 1

Only when all the following conditions are met can the job enter a business-complete state:

Condition	Meaning
`missing = ∅`	No missing records
`unknown = ∅`	No unknown records
`duplicate = ∅`	No duplicate records
`	E

The provider’s job status can only serve as a signal that “final reconciliation can begin,” not a substitute for these set conditions.

The reconciler must parse both success and failure channels:

def reconcile(expected_ids, output_rows, error_rows):
    success = index_unique(output_rows, key="record_id")
    failed  = index_unique(error_rows,  key="record_id")
    seen = set(success) | set(failed)
    missing   = expected_ids - seen
    unknown   = seen - expected_ids
    duplicate = find_duplicates(output_rows + error_rows)
    return {
        "success": success,
        "failed": failed,
        "missing": missing,
        "unknown": unknown,
        "duplicate": duplicate,
    }

Provider	Success Channel	Failure Channel
OpenAI	Output file + success count	Error file + failure count
Amazon Bedrock	`modelOutput` + `manifest.json.out`	`error` field
Gemini	Normal response	Error response or expired with no result

Shard Retry: Retry Records, Not Jobs

Failed records should first be categorized:

Category	Description	Handling
Permanent failure	Schema error, content limit exceeded, permission denied	Manual handling
Retryable failure	Timeout, transient error, capacity insufficient	Include in retry shard
Unqualified result	Response succeeded but fails business quality gate	Include in retry shard
Missing record	Input exists but no record in any output channel	Include in retry shard

The new retry shard contains only retryable failures, unqualified results, and missing records, and preserves the correlation fields:

{
  "record_id": "stable-business-id",
  "attempt": 2,
  "parent_job_key": "sha256:...",
  "parent_shard_id": "part-00017",
  "retry_reason": "request_timeout"
}

Do not generate completely unrelated random record IDs for each retry, otherwise the system cannot determine that this is a re-execution of the same business record.

Retries should also set:

Maximum attempt count: to prevent infinite retries
Exponential backoff or next batch window
Error code whitelist: only retry known recoverable errors
Per-record cumulative token and cost limits
Manual handling queue: for records exceeding limits
Output contract validation when falling back to a different provider

Cancellation, Expiration, and Partial Results

Cancellation does not mean “no results.”

Provider	Cancellation/Expiration Behavior
OpenAI	Completed parts may still appear in the output file
Amazon Bedrock	Processed tokens are still billed
Gemini	Distinguishes between cancellation and expiration; expired jobs may have no retrievable results

Correct actions after cancellation:

Freeze the window for new result writes.
Download existing output and error files.
Run a full reconciliation.
Continue to validate successfully completed records; do not resubmit them.
Place only missing or retryable failures into a compensation batch.
Record the cancellation time, processed count, and cost snapshot.

Cost and Audit: Costs Must Be Tracked to Records and Attempts

Batch interface discounts cannot replace cost governance. Each result should record:

Field	Description
`record_id`	Stable business ID
`attempt`	Attempt number
`provider_job_id`	Provider job ID
`provider_request_id`	Provider request ID
Input/Output/Cache Tokens	Usage details
Model version returned by provider	Exact version string
Status	Success, failure, cancelled, or expired
Estimated per-record cost	For cost attribution
Result file and raw response hash	Audit trail

Cost reports should at least distinguish:

First-time success cost
Retryable failure consumption
Duplicate submission cost
Quality gate failure cost
Manual compensation cost

Only by knowing “how many times the same record was executed” can you truly identify duplicate billing in batch jobs.

Applicable Scenarios

This approach is suitable for:

Large-scale offline content extraction and classification
Nightly evaluation and security regression
Historical data summarization and structured backfill
Embedding or multimodal feature batch generation
Data cleaning, synthetic data, and training set construction

Not suitable for online requests requiring sub-second response, strong interactivity, or immediate failure feedback.

Common Misconceptions

Using Batch ID as the Sole Idempotency Key

The Batch ID is only returned after the provider successfully creates the job. It cannot protect against the window where “the server created the job but the client didn’t receive the response.”

Assuming Output Order Equals Input Order

Some platforms explicitly do not guarantee order. Even if a platform currently maintains order, you should correlate by a stable record ID to allow for future migration.

Directly Overwriting Business Tables on Job Success

You must first complete record-level set reconciliation, schema validation, and quality gates before publishing results.

Rerunning the Entire Batch for a Single Failure

This consumes tokens for already-successful records. The correct approach is to generate a retry shard containing only the failed subset.

Saving Only the Final Text, Not the Raw Response

Without the raw response, request ID, usage, and error code, you cannot audit, dispute charges, or reproduce issues.

Using Random UUIDs, Unable to Identify Business Duplicates

Random UUIDs guarantee uniqueness but cannot express the stable identity of the same business input. Use deterministic Record IDs and a separate Attempt ID.