If temperature is set to 0, why can the same prompt still yield different answers?

temperature=0 only disables sampling randomness. Dynamic batching, request scheduling, floating-point reduction order, parallelism, kernel selection, and software/hardware version changes can still cause subtle numerical differences in close logits, leading to a fork at a particular token position.

Is fixing the seed sufficient for reproducible inference?

No. A seed only constrains the random number generator; it cannot eliminate issues from missing Batch Invariance, different reduction orders, different GPUs, or different inference framework versions.

Does enabling Batch Invariance degrade inference performance?

It can. Fixing reduction strategies and kernel configurations reduces optimization opportunities for different shapes. It's recommended to apply strict mode only to high-value traffic and measure throughput and latency costs through stress testing.

Should we pursue bitwise consistency across different GPU models?

Unless your business absolutely requires it, the cost is usually too high. A more practical goal is to first achieve request-level reproducibility within the same hardware and version, then use replay evaluations to ensure quality compatibility across environments.

LLM Inference Reproducibility in Production: Controlling Output Drift with Batch Invariance, Fixed Seeds, and Configuration Fingerprints

Why temperature=0 Still Doesn’t Mean “Same Input, Same Output”

Many teams attribute the randomness of large language models solely to sampling parameters: set temperature to 0, disable top_p, and fix a seed, and you should get identical outputs, right?

This understanding only covers the most superficial layer of randomness.

temperature=0 typically means selecting the token with the highest probability at each step—greedy decoding. But in real inference services, requests are dynamically batched, with batch sizes fluctuating with traffic; the inference framework selects different kernels based on input shape, sequence length, and parallelism; and the order of floating-point reductions in matrix multiplication, normalization, and attention can also change. If the logits of two candidate tokens are close enough, a tiny numerical difference can alter the first choice, and the entire generation path diverges from there.

Therefore, inference reproducibility is not a sampling parameter problem; it’s a systems engineering problem spanning model artifacts, decoding configurations, schedulers, kernels, hardware, and runtime environments.

An analysis by Thinking Machines Lab in 2025 further pointed out: common LLM forward kernels can be deterministic per single run, but they may not possess Batch Invariance—when the same sample is placed in batches of different sizes or shapes, the reduction strategy can change, leading to different numerical results. Online concurrency determines batch shape, and concurrency is uncontrollable from a single user’s perspective, making the service appear non-deterministic to the user.

First, Distinguish Four Types of “Output Inconsistency”

Before production governance, you must separate drifts from different sources; otherwise, you’ll end up blaming everything on the seed.

1. Sampling Randomness

When temperature > 0, nucleus sampling, or other random sampling is enabled, different results are part of the distribution. Fixing the seed can control this randomness under a specific implementation and fixed call order, but it does not guarantee reproducibility across processes, versions, or hardware.

2. Run-to-Run Non-Determinism of Kernels

Some GPU operations may use atomic additions or reductions without a fixed execution order. Even with identical inputs run consecutively, differences in the least significant bits can appear. For typical LLM forward passes, this may not be the primary source, but attention is needed for custom operators, third-party kernels, and specific quantization implementations.

3. Batch and Scheduling Non-Invariance

This is the most overlooked source in online serving. The same request in batch sizes 1, 8, and 32 may use different tile sizes, Split-K, or Split-KV strategies for matrix multiplication or attention. Chunked prefill, dynamic batching, and request preemption also change the computational partitioning of sequences. As long as numerical results depend on batch or partitioning, traffic fluctuations translate into output variations.

4. Environment and Version Differences

GPU model, CUDA, drivers, cuBLAS, NCCL, PyTorch, vLLM, quantization backends, tensor parallel size, and compilation parameters can all alter the computation path. Even if a configuration is reproducible on a single machine, you cannot directly infer bitwise reproducibility across different hardware or versions.

NVIDIA’s floating-point documentation explicitly states that operation order, thread count, FMA usage, and parallel reduction structure affect the final numerical values. vLLM’s reproducibility documentation also notes that the default configuration prioritizes performance and does not guarantee reproducible results; even with relevant settings enabled, the guarantee is limited to the same hardware and the same vLLM version.

Core Principles: What Seeds, Deterministic Scheduling, and Batch Invariance Each Solve

Fixed Seed: Controls Random Numbers, Not Computation Shape

A seed fixes the state of the random number generator in Python, NumPy, PyTorch, or the inference framework. It’s suitable for controlling sampling, but it cannot guarantee:

Dynamic batches remain the same;
The same kernel is used each time;
The reduction tree for tensor parallelism remains the same;
Numerical values remain unchanged after upgrading CUDA or the framework;
The same request produces consistent output under different concurrent loads.

So, a fixed seed is a necessary condition, but not a sufficient one.

Deterministic Scheduling: Fixes How Requests Enter the Computation Graph

Offline evaluations can disable multi-process scheduling, fix the batch, and fix input ordering, ensuring each run uses the same execution plan. One offline approach documented by vLLM is to disable V1 Multiprocessing for deterministic scheduling. In online mode, due to constantly changing real traffic, relying solely on fixed scheduling is usually infeasible.

Batch Invariance: Makes a Single Request Independent of Other Requests in the Same Batch

The goal of Batch Invariance is not simply to fix the batch size, but to make the numerical result of a request independent of:

How many other requests are processed simultaneously;
The position of the request within the batch;
How many segments the prefill is split into;
How many Split-KV partitions are used during decode.

Implementation typically requires fixing the reduction order, unifying kernel configurations, or using specially designed Batch-Invariant RMSNorm, Matmul, and Attention. The cost is sacrificing some extreme optimizations for specific shapes, so it should be enabled based on business value, not enforced globally.

Engineering Implementation: Establishing Three Tiers of Reproducibility

Making “determinism” a single on/off switch is often too expensive. A more practical approach is to establish three service tiers.

Tier	Use Case	Core Requirements
Level 0: Statistical Consistency	General chat, creative generation, non-critical summarization	Stable quality, refusal rate, length distribution, and safety metrics; no per-token consistency required
Level 1: Request-Level Reproducibility	Regression testing, prompt debugging, cache building, issue post-mortems	Consistent token sequences under the same artifact, environment, parallelism, and decoding parameters
Level 2: Strict Determinism	Compliance audits, automated decisions, RL rollout alignment, bitwise comparison experiments	Batch-Invariant kernels, fixed hardware and software stack, dedicated deterministic service pool

Level 0: Statistical Consistency

Keep normal dynamic batching and high-performance kernels online. Require stable quality, refusal rate, length distribution, and safety metrics on a fixed test set, but do not require per-token consistency.

Level 1: Request-Level Reproducibility

Require that the same request, when executed repeatedly under the same model artifact, environment, parallelism, and decoding parameters, produces an identical token sequence. This can be achieved by fixing the seed, scheduling, batch buckets, and kernel configurations.

Level 2: Strict Determinism

Require enabling Batch-Invariant kernels, fixing the hardware and software stack, and imposing strict constraints on TP size, reduction order, and attention partitioning. This tier should be placed in a dedicated deterministic service pool to prevent regular traffic from altering the scheduling shape.

Configuration Fingerprints: Without an Environment Snapshot, You Can’t Explain Reproducibility Failures

Every inference should generate an inference_fingerprint. This is not simply recording the model name, but hashing the complete execution contract.

{
  "model_sha256": "...",
  "tokenizer_sha256": "...",
  "chat_template_sha256": "...",
  "dtype": "bfloat16",
  "quantization": "none",
  "sampling": {
    "temperature": 0,
    "top_p": 1,
    "seed": 42
  },
  "runtime": {
    "vllm_version": "pinned-version",
    "pytorch_version": "pinned-version",
    "cuda_version": "pinned-version",
    "driver_version": "pinned-version"
  },
  "hardware": {
    "gpu_model": "pinned-gpu",
    "tensor_parallel_size": 1
  },
  "scheduler": {
    "batch_invariance": true,
    "chunked_prefill": false,
    "max_num_seqs": 16
  }
}

The fingerprint must be logged together with the output, token sequence, finish reason, request ID, and deployment version. When discrepancies arise, the system first compares fingerprints, rather than immediately suspecting the model is “occasionally misbehaving.”

Replay Matrix: Don’t Just Run Ten Times in an Idle Environment

Many “reproducibility tests” simply call the model ten times serially on a development machine. This fails to uncover batch-sensitive issues in production.

At a minimum, cover the following dimensions:

Dimension	Suggested Test Values
Concurrency	1, 8, 32, near peak
Prompt Length	Short, medium, near context limit
Output Length	16, 128, long output
Batch Shape	Uniform length, mixed lengths
Prefill Strategy	Full prefill, chunked prefill
Parallelism	TP=1 vs. production TP configuration
Runtime Environment	Current version vs. version to be upgraded
Restart Boundary	Same process, after restart, after redeployment

For each sample, record the following metrics:

Exact Token Match: Is the complete token sequence identical?
First Divergence Position: At which token does the first fork occur?
Top-K Agreement: Is the set of candidate tokens stable before and after the fork?
Logit Margin: The difference between the top-1 and top-2 logits, identifying naturally fragile samples.
Semantic Agreement: Only for Level 0; cannot replace strict token comparison.

A simple replay checker can be implemented as follows:

from dataclasses import dataclass

@dataclass(frozen=True)
class ReplayResult:
    token_ids: list[int]
    fingerprint: str

def first_divergence(a: list[int], b: list[int]) -> int | None:
    for index, (left, right) in enumerate(zip(a, b)):
        if left != right:
            return index
    if len(a) != len(b):
        return min(len(a), len(b))
    return None

def assert_reproducible(results: list[ReplayResult]) -> None:
    baseline = results[0]
    for current in results[1:]:
        if current.fingerprint != baseline.fingerprint:
            raise ValueError(
                "Execution fingerprint changed; results are not directly comparable"
            )
        divergence = first_divergence(baseline.token_ids, current.token_ids)
        if divergence is not None:
            raise AssertionError(
                f"Token output diverged at position {divergence}"
            )

Online Architecture: Separate Deterministic Traffic into a Dedicated Pool

The safest architecture is not to force all requests into the strictest mode, but to add a reproducibility_tier for routing:

General generation requests     ──> High-throughput dynamic batching pool
Regression and audit requests   ──> Fixed-configuration reproducible pool
Strict deterministic requests   ──> Batch-Invariant dedicated pool

The gateway is responsible for verifying:

Whether a fixed seed is provided;
Whether the model and tokenizer fingerprints match;
Whether the request is routed to the correct hardware pool;
Whether the current deployment allows chunked prefill, speculative decoding, or dynamic kernels;
Whether the results need to save the full token sequence and logprob evidence.

This approach preserves throughput for regular traffic while providing a stable execution environment for evaluations and audits.

Applicable Scenarios

The most valuable scenarios for inference reproducibility include:

Model and prompt regression testing: Avoid misinterpreting system noise as quality degradation.
Offline evaluations and leaderboards: Ensure different versions are compared under the same execution contract.
Compliance audits: Be able to explain which artifact and environment generated a particular output.
Result caching: Confirm that the cache key covers not only the prompt but also the execution fingerprint.
Agent automation: Stable post-mortem analysis for critical tool parameter generation.
RL rollout: Reduce bias from numerical inconsistencies when training and inference parallelism strategies change.

Common Misconceptions

Misconception 1: temperature=0 Means Determinism

It only eliminates the primary randomness from the sampling layer, not differences from dynamic batching and numerical paths.

Misconception 2: A Fixed Seed Guarantees Cross-Version Reproducibility

A seed does not fix CUDA kernels, GPU instructions, quantization backends, or reduction trees. You must re-run the replay matrix after upgrading any layer.

Misconception 3: Semantically Equivalent Answers Count as Full Reproducibility

This may be acceptable for chat applications, but for regression pinpointing, caching, audits, and RL, token-level differences still matter. First define the service tier, then decide on the comparison standard.

Misconception 4: Disabling Dynamic Batching Is the Only Solution

Disabling dynamic batching simplifies offline reproducibility but significantly sacrifices online throughput. A dedicated deterministic pool or Batch-Invariant kernels are more suitable for production environments.

Misconception 5: Only Record the Model Name

The same model name can correspond to different weights, tokenizers, chat templates, quantization configurations, and inference versions. Without a complete fingerprint, it’s nearly impossible to locate the cause of a reproducibility failure.

Go-Live Checklist

References

Thinking Machines Lab, Defeating Nondeterminism in LLM Inference — thinkingmachines.ai
vLLM Documentation, Reproducibility — docs.vllm.ai
NVIDIA, Floating Point and IEEE 754 — docs.nvidia.com
Zhang et al., Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch — arxiv.org
Gond et al., LLM-42: Enabling Determinism in LLM Inference with Verified Speculation — arxiv.org