Does setting temperature to 0 guarantee identical outputs?

No. Greedy decoding can still change the first token due to dynamic batching, parallel reduction order, kernel selection, and low-precision numerical errors that alter critical logit rankings.

Why might online requests with the same seed still produce different results?

A seed only constrains the random number sequence, not the batch composition, floating-point reduction order, parallel topology, inference backend version, or kernel implementation. Any change in these factors can introduce numerical drift.

Should deterministic mode be enabled for all production requests?

Generally, no. Deterministic mode may sacrifice throughput, latency, and available optimizations. It's better suited for evaluation, regression testing, auditing, RL rollouts, and high-risk business channels. Consider offering dual channels: a high-performance channel for normal requests and a deterministic channel for critical traffic.

Why is the same request stable under low concurrency but occasionally different under high concurrency?

High concurrency changes request composition, batch size, execution shape, and reduction paths. If the inference engine lacks batch invariance, the same request can experience numerical drift due to different co-located requests in the same batch.

LLM Inference Reproducibility in Production: Controlling Output Drift with Seed Contracts, Batch Invariance, and Deterministic Kernels

Background: Same Seed ≠ Same Output

Many teams reduce LLM inference reproducibility to three conditions: identical model weights, identical prompt, and identical seed. This only covers sampling randomness, not the numerical and scheduling variations within the inference system itself.

Online inference typically involves dynamic batching, continuous scheduling, multiple CUDA streams, tensor parallelism, low-precision computation, fused operators, and automatic kernel selection. Floating-point addition is non-associative; summing the same set of numbers in different orders can produce subtle differences. Most of the time, these differences don’t change the answer. But when two candidate tokens have very close logits, a tiny numerical perturbation can alter the first token, and subsequent autoregressive generation quickly diverges.

Therefore, inference reproducibility is not a boolean switch but a set of engineering contracts that need explicit constraints.

First, Define Your Reproducibility Tier

Before deployment, determine which level of stability your business actually needs to avoid unnecessary performance costs for “absolute consistency.”

Tier	Definition	Use Cases
Semantic Consistency	Output text may differ, but conclusions, structure, and key facts are consistent	General Q&A, creative writing, summarization
Token-Level Consistency	Identical requests must produce exactly the same token sequence	Regression testing, automated evaluation, cache key validation, audit replay, RL rollout
Numerical Tolerance Consistency	Does not require token identity, but per-step logit, Top-K ranking, or probability distribution differences are within a threshold	Diagnosing numerical drift from kernel, quantization, or parallel topology changes
Bitwise Consistency	Output tensors are identical bit-for-bit	Only meaningful with fixed hardware, fixed software versions, fixed execution paths, and fixed parallel configurations

Where Output Drift Comes From

1. Incomplete Request Contracts

Recording only the prompt and seed is insufficient. Production replay must freeze at least the following fields:

Model and weight artifact fingerprint;
Tokenizer, chat template, and special token versions;
Sampling parameters: temperature, top_p, top_k, min_p, repetition penalty, etc.;
Seed scope: is it per-request independent seed or a global worker random state?
Logits processor, stop words, and max output length;
Inference backend, quantization format, and attention backend.

Missing any one of these can make the “same request” a different computational task from the system’s perspective.

2. Dynamic Batching Changes Execution Shape

Online schedulers combine requests into different batches. The same request may execute alongside short requests, long requests, or requests at different stages, leading to changes in matrix shape, padding, reduction paths, and kernel selection.

The vLLM documentation explicitly states that default configurations prioritize performance and do not guarantee reproducible results. The latest documentation offers Batch Invariance, which makes output independent of batch size and request order within a batch, but this feature is still in Beta and may incur a performance penalty.

3. CUDA Kernel and Floating-Point Reduction Order Changes

Even with greedy decoding, GEMM, attention, all-reduce, and fused operators in the computation graph can produce numerical deviations due to different execution orders.

PyTorch documentation notes that fully reproducible results are not guaranteed across PyTorch versions, commits, platforms, or CPU/GPU environments. torch.use_deterministic_algorithms(True) forces the use of known deterministic implementations and raises errors when none are available, but it typically reduces performance.

cuBLAS provides some bitwise reproducibility guarantees for the same toolkit version, GPU architecture, and SM count. However, this guarantee may break when multiple CUDA streams run concurrently. Official recommendations include using per-stream independent workspaces, independent handles, or setting CUBLAS_WORKSPACE_CONFIG to reduce nondeterminism.

4. Parallel Topology and Backend Version Changes

Tensor parallelism count, collective reduction trees, custom all-reduce, quantization kernels, and CUDA graph capture paths all affect computation order.

Recent research further indicates that even with identical model weights, hardware, and decoding parameters, different inference backends can produce significant output and evaluation differences. This conclusion comes from specific model and benchmark experiments and cannot be generalized to all systems, but it’s sufficient to show: the inference backend itself should be treated as a versioned variable in evaluation and regression.

Engineering Implementation: Establish a Deterministic Dedicated Channel

It’s not recommended to force all online traffic into deterministic mode. A more sensible approach is to maintain two execution channels:

High-Performance Channel: Serves normal production requests, allowing dynamic batching and high-performance kernels;
Deterministic Channel: Serves evaluation, regression, auditing, issue replay, and high-risk requests, freezing scheduling, kernels, and environment.

This limits the performance cost to traffic that truly needs reproducibility.

Step 1: Define the Request Reproducibility Envelope

Every replayable request should save an immutable execution envelope:

{
  "request_id": "req-20260714-001",
  "model_artifact_sha256": "...",
  "tokenizer_sha256": "...",
  "chat_template_sha256": "...",
  "prompt_token_ids": [128000, 2028, 374],
  "sampling": {
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256,
    "seed": 42
  },
  "runtime": {
    "engine": "vllm",
    "engine_version": "pinned-version",
    "torch_version": "pinned-version",
    "cuda_version": "pinned-version",
    "gpu_model": "pinned-model",
    "tensor_parallel_size": 1,
    "attention_backend": "pinned-backend",
    "quantization": "none"
  }
}

Prefer saving Prompt Token IDs over raw strings. This isolates tokenizer and template changes from numerical inference issues.

Step 2: Enable Batch Invariance or Fix Scheduling

According to the latest vLLM documentation, to get batch-independent output in online serving, enable Batch Invariance:

VLLM_BATCH_INVARIANT=1 \
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
vllm serve /models/your-model \
  --tensor-parallel-size 1

Still pass the seed explicitly in the call:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://127.0.0.1:8000/v1",
)

response = client.completions.create(
    model="your-model",
    prompt="Explain floating-point non-associativity.",
    temperature=0.7,
    top_p=0.95,
    max_tokens=256,
    seed=42,
)
print(response.choices[0].text)

Note: Batch Invariance currently has boundaries in hardware, model coverage, and performance. You must validate it on your own model, GPU, tensor parallel configuration, and concurrency mode before deployment. Don’t assume that setting the environment variable alone guarantees determinism.

Step 3: Freeze Kernel and Parallel Configuration

The deterministic channel should pin the following:

GPU model, compute capability, and driver version;
CUDA, cuBLAS, cuDNN, PyTorch, and inference engine versions;
Tensor parallelism, pipeline parallelism, and data parallelism counts;
Attention backend, all-reduce implementation, and CUDA graph strategy;
Quantization format, scale files, and quantization kernels;
CUDA stream, workspace, and worker counts.

For custom PyTorch inference programs, enable in the test channel:

import os
import random
import numpy as np
import torch

os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.use_deterministic_algorithms(True, warn_only=False)
torch.backends.cudnn.benchmark = False

Set environment variables before the Python process starts. Deterministic settings may cause operator errors or performance degradation, so perform compatibility scanning during image building and pre-release stages.

Step 4: Build a Multi-Dimensional Replay Matrix

Running the same request ten times in a row doesn’t cover the main sources of production drift. The replay matrix should include at least:

Dimension	Variation
Batch	Single request, fixed small batch, high-concurrency dynamic batch
Order	Different request arrival orders
Input	Different input length combinations
CUDA Graph	Enabled / Disabled
Attention Backend	Switching between backends
Tensor Parallel	Different TP configurations
Version	Parallel replay of old and candidate versions

For each group of requests, record at least:

Whether the token sequence is identical;
The position of the first diverging token;
The Top-K tokens and logprobs at the step before divergence;
The maximum absolute and relative logit differences;
The semantic and business metrics of the final text;
Throughput, P95 latency, and memory changes.

The first divergence point is more diagnostic than “whether the final answer is the same.” Divergence at the first token typically points to input, template, or forward computation differences; divergence later in a long output is more likely due to gradual accumulation of tiny numerical errors.

Step 5: Include Environment Fingerprints in Release Gates

Beyond model artifacts, generate fingerprints for the inference environment. Include at least:

model_sha256
tokenizer_sha256
chat_template_sha256
engine_version
torch_version
cuda_version
cublas_version
gpu_name_and_compute_capability
attention_backend
quantization_method
tensor_parallel_size
batch_invariance_flag
sampling_contract_version

Any field change should trigger replay, rather than waiting for “occasional different answers” in production.

Applicable Scenarios

Inference reproducibility governance is particularly suitable for:

Model, inference backend, and CUDA kernel upgrade regression;
LLM automatic evaluation and benchmark re-testing;
Reinforcement learning rollout and training-inference consistency checks;
High-risk workflows requiring audit replay in finance, insurance, healthcare;
Agent workflow failure replay;
Cache, idempotency, and digital signature verification;
Multi-vendor or multi-backend result discrepancy diagnosis.

General chat and creative writing scenarios typically don’t need bitwise consistency; semantic stability through business evaluation is sufficient.

Common Misconceptions

Myth 1: temperature=0 is deterministic. temperature=0 usually means greedy selection, but logits themselves can still change due to batching and kernels. When two token scores are close, tiny errors can still change the selection.

Myth 2: Seed is a complete reproducibility credential. A seed only constrains the random number state. Without fixed token IDs, backend versions, batch behavior, parallel topology, and kernels, the seed only solves part of the problem.

Myth 3: Bitwise consistency should hold across GPUs and versions. PyTorch, cuBLAS, and vLLM documentation all set clear boundaries on reproducibility scope. Bitwise consistency across versions, platforms, and hardware should not be a default promise.

Myth 4: Enabling deterministic mode everywhere is safest. Deterministic implementations may reduce throughput, disable some optimizations, or even not support certain operators. Production systems should provide deterministic channels by business tier, not sacrifice performance indiscriminately.

Myth 5: Only compare final text. Identical final text doesn’t mean numerical stability; different final text doesn’t necessarily mean business quality degradation. Check tokens, first divergence point, logprobs, and business metrics simultaneously.

Deployment Checklist

FAQ

Can I enable determinism only during evaluation and disable it in production serving?

Yes, and it’s often more reasonable. However, be aware that there are execution differences between the evaluation and production channels. For critical versions, supplement with shadow replay under real production configurations to avoid evaluation results that don’t represent online behavior.

What if the vendor API doesn’t provide a seed or environment version?

In this case, token-level or bitwise reproducibility cannot be promised. Save the request, response, model alias, timestamp, and any version identifiers returned by the vendor. Use multiple sampling, semantic metrics, and business assertions for statistical regression.