LLM Model Format Conversion in Production: Avoiding Silent Degradation with Tensor Mapping, QKV Reordering, and Layer-Wise Validation

Introduction: Why a Model That “Loads” Can Still Be Wrong

Model format conversion is often misunderstood as simply swapping .bin for .safetensors or reshuffling weight files. For large models, the real object of conversion is not files but the complete model semantic contract: parameter names, tensor shapes, dimension semantics, attention head layout, parallel slicing strategy, shared weight relationships, tokenizer, and configuration must all remain consistent.

The most dangerous failure mode of a conversion script is not an immediate error, but silent degradation. The model loads, and may even generate text, but output quality, numerical stability, or specific capabilities have changed. Typical causes include:

• Independent q_proj, k_proj, v_proj incorrectly fused into a QKV tensor.
• GQA model reordered using standard MHA head counts, causing KV head misalignment.
• SwiGLU’s gate_proj and up_proj concatenated in the wrong order.
• Row Parallel and Column Parallel using the wrong slicing dimension.
• Pipeline Parallel export missing parameters not belonging to the current stage.
• Embedding and LM Head originally shared, becoming two separate weight sets after conversion.
• config.json settings for head count, RoPE, vocabulary size, or tie-weight inconsistent with weights.
• Shard files exist, but model.safetensors.index.json’s weight_map points to the wrong shard.

Therefore, a production conversion pipeline must demote “loadable” to the lowest level of check, and establish structural coverage, tensor statistics, intermediate layer outputs, and final logits alignment as formal acceptance criteria.

Core Principles: Conversion as a Four-Layer Contract Mapping

Layer 1: Configuration Semantic Mapping

Model configuration is the starting point of conversion. Freeze and compare the following fields:

Configuration Item	Example
hidden_size, num_hidden_layers	4096, 32
num_attention_heads, num_key_value_heads	32, 8 (GQA)
intermediate_size and activation function	11008, SwiGLU
RoPE parameters, max context length	theta=500000, 8192
Vocabulary Size, special token IDs	32000, BOS/EOS
Whether Embedding and LM Head are shared	tie_word_embeddings=true
MoE expert count, Top-K, and Expert Parallel layout	8 experts, top-2
Target Dtype and quantization scale tensors	BF16, FP8

Configuration should not be “guessed from shapes” by the conversion script. Production pipelines should save source configuration fingerprints, target configuration fingerprints, and explicit mapping rules. Any undeclared change should block release.

Layer 2: Parameter Name Mapping

Different frameworks use different names for the same layer. For example, in Hugging Face:

model.layers.0.self_attn.q_proj.weight
model.layers.0.self_attn.k_proj.weight
model.layers.0.self_attn.v_proj.weight

Megatron might use a fused version:

decoder.layers.0.self_attention.linear_qkv.weight

Avoid scattering string replacements throughout the code. Establish a versioned Tensor Mapping Manifest that explicitly defines source parameters, target parameters, transformation functions, target shapes, dtypes, parallel assignments, and inverse transformation rules:

{
  "target": "decoder.layers.*.self_attention.linear_qkv.weight",
  "sources": [
    "model.layers.*.self_attn.q_proj.weight",
    "model.layers.*.self_attn.k_proj.weight",
    "model.layers.*.self_attn.v_proj.weight"
  ],
  "transform": "qkv_interleave_gqa",
  "tp_axis": 0,
  "required": true
}

The manifest must be part of the artifact version and code review. After conversion, the system should output: total source parameters, consumed parameters, total target parameters, unmapped parameters, double-consumed parameters, and a list of allowed exceptions.

Layer 3: Structural Transformations

The most common source of silent errors is one-to-many and many-to-one mappings.

QKV Fusion and Splitting must understand the actual layout of the target framework, not simply execute torch.cat([q, k, v]). Some implementations arrange tensors interleaved by head; in GQA, the number of Query heads differs from the number of KV heads, so reordering logic must use the model configuration to compute each head’s interval.

Gated MLP often merges Gate and Up Projections, but different frameworks may use [gate, up] or another internal order. An incorrect order usually doesn’t trigger shape errors but significantly alters activation results.

Tensor Parallel requires distinguishing Column Parallel from Row Parallel. The former typically slices along the output dimension, the latter along the input dimension. Conversion must first complete structural reordering, then slice according to the target layout, or strictly follow the inverse operations defined by the target framework.

Pipeline Parallel and Expert Parallel also need to handle parameter ownership: a rank lacking a local tensor does not mean the parameter doesn’t exist. During export, parameters must be collected from the owning stage and written in a deterministic global order.

Layer 4: File and Shard Mapping

Safetensors provides safe, fast tensor storage, but it won’t validate model semantics for you. Large models are typically split into multiple shards, with a model.safetensors.index.json weight_map mapping parameter names to specific files.

Production conversion should generate its own manifest, recording at least:

• File size and digest of each shard.
• Name, shape, dtype, and byte range of each tensor.
• Total parameter count and total byte count.
• Source model, converter code, and dependency versions.
• Conversion configuration and parallel layout.
• Completion marker and atomic commit status.

Don’t let consumers see a half-written target directory. It’s recommended to write to a temporary directory first, then write a commit marker or atomically switch an alias after validation is complete.

Engineering Implementation: Establishing a Five-Level Conversion Gate

Level 1: Configuration and Parameter Coverage Gate

Before and after conversion, compare configuration fields and parameter sets:

• All Required Source Tensors must be consumed exactly once.
• All Required Target Tensors must be generated.
• Missing, Unexpected, and Duplicate Mappings block by default.
• Allowed missing items must use an explicit whitelist with an expiration date and an owner.
• Shape and dtype changes must be declared in the manifest.

“Ignore all missing keys” should not be the production default. Lenient loading is suitable for exploration, not release.

Level 2: Tensor Statistics and Summary Gate

For directly mapped tensors, compare summaries and statistics; for tensors that have been fused, split, or transposed, compare after applying the inverse transformation.

It is recommended to save:

Metric	Description
Shape, dtype, element count	Basic consistency
min, max, mean, std	Distribution overview
L1/L2 Norm	Overall scale
NaN/Inf count	Anomaly detection
Chunked or sampled summary	Local consistency

Don’t compare only a single mean across the entire tensor. Two completely different weight sets can have similar means and variances. Summaries are for quick problem detection, not a substitute for numerical alignment.

Level 3: Layer-Wise Forward Alignment

Use fixed token IDs as input, run both source and target models simultaneously, and register hooks at critical boundaries:

• Embedding output.
• Attention output per layer.
• MLP output per layer.
• Final norm.
• LM head logits.

Find the first layer exceeding tolerance; this localizes issues far more efficiently than comparing only final text. Tolerance must be calibrated by dtype, kernel, and backend; don’t copy a single global constant.

from __future__ import annotations
import torch

def compare_tensor(name: str, source: torch.Tensor, target: torch.Tensor) -> dict[str, float]:
    if source.shape != target.shape:
        raise ValueError(f"{name}: shape mismatch {source.shape} != {target.shape}")
    source_fp32 = source.detach().float().cpu()
    target_fp32 = target.detach().float().cpu()
    diff = (source_fp32 - target_fp32).abs()
    return {
        "max_abs": diff.max().item(),
        "mean_abs": diff.mean().item(),
        "source_norm": source_fp32.norm().item(),
        "target_norm": target_fp32.norm().item(),
    }

def assert_finite(name: str, tensor: torch.Tensor) -> None:
    if not torch.isfinite(tensor).all():
        raise FloatingPointError(f"{name}: non-finite values detected")

During comparison, disable dropout, fix the input and execution environment, and record model, CUDA, framework, kernel, and parallel configuration versions.

Level 4: Logits and Ranking Consistency Gate

Final logits checks should include at least:

• Fixed short input.
• Multi-language and special tokens.
• Input near the maximum context length.
• Use cases related to GQA, RoPE, or MoE features.
• Per-position logits error.
• Top-K token set and order.
• Position of the first diverging token.

Generated text is retained only as a supplementary check. Sampling amplifies small differences, and identical text can mask deeper logits drift.

Level 5: Round-Trip and Task Regression Gate

For bidirectional conversion pipelines, execute:

Source Format → Target Format → Source Format

After the round-trip, compare configuration, parameters, layer-wise outputs, and logits again. Then validate with a small task set: perplexity, basic generation, key business capabilities, and performance baselines.

Round-trip does not prove absolute correctness, as both directions might share the same error; but it effectively discovers irreversible mappings, missing parameters, and incorrect sharding.

Release Process Design

Treat each conversion as a controlled build, not a manual script execution:

1. Lock the source model revision, configuration, tokenizer, and weight digest.
2. Lock the converter Git commit, dependencies, and target framework version.
3. Generate a mapping plan, manually review high-risk one-to-many and many-to-one rules.
4. Execute streaming conversion in an isolated directory.
5. Run parameter coverage, tensor statistics, layer-wise forward, and logits replay.
6. Output a machine-readable conversion report.
7. On success, write a commit marker and publish an immutable artifact version.
8. Shadow-load the target artifact, verify startup time, memory, throughput, and business regression.
9. Gradual rollout, retaining the source artifact and a fast rollback entry point.

The conversion report should answer: who converted, from which revision, using what mapping rules, which parameters underwent structural transformations, what tolerance was used, which checks passed, and what the final artifact digest is.

Applicable Scenarios

This governance approach applies to:

• Importing Hugging Face models into Megatron for large-scale training.
• Exporting Megatron training results back to Hugging Face or vLLM for deployment.
• Converting from Hugging Face, NeMo, etc., to TensorRT-LLM checkpoints.
• Changing Tensor Parallel, Pipeline Parallel, or Expert Parallel world sizes.
• Converting between independent Q/K/V and fused QKV.
• Reorganizing expert parameters for dense and MoE models.
• Migrating formats for BF16, FP16, FP8, or quantization scale tensors.
• Integrating custom models into a unified inference engine.

Common Misconceptions

Misconception	Risk
Only checking parameter count	Consistent parameter count doesn’t mean correct mapping. Two tensors may have the same shape but belong to different layers, projections, or head arrangements.
Only comparing generated text	Identical text may just be insensitive to the current prompt; different text may come from backend non-determinism. First compare layer-wise outputs and logits under fixed input.
Directly allowing missing keys	Allowing missing keys turns structural errors into random initialization or default values. Production should default to strict, with auditable exceptions.
Treating shards as ordinary slices	Shards from different parallel frameworks may contain interleaved heads, expert groupings, or stage assignments. Don’t just concatenate by file index.
Ignoring configuration and tokenizer	Correct weight conversion with wrong configuration still produces degradation. Configuration, tokenizer, chat template, and special tokens must be released together with weights.
Overwriting in the target directory	A failed or interrupted conversion leaves a half-finished artifact. Use a temporary directory, complete validation, and atomic commit.

Go-Live Checklist

Artifacts and Configuration

• Source revision, converter commit, and target version are frozen.
• Configuration fields have explicit source-to-target mappings.
• Tokenizer, special tokens, and vocabulary size are consistent.
• Shard index is bidirectionally consistent with actual files.
• Artifact digest, conversion report, and commit marker are complete.

Parameter Conversion

• Required Source Tensors are all consumed exactly once.
• Required Target Tensors are all generated.
• QKV, Gated MLP, Embedding, LM Head, and MoE mappings are individually validated.
• TP/PP/EP slicing dimensions and rank assignments are correct.
• Unmapped parameters and allowed exceptions have audit records.

Numerical Validation

• Tensor shape, dtype, and NaN/Inf checks pass.
• Alignment of directly mapped and inverse-transformed tensors passes.
• Layer-wise forward finds no anomalous divergence point.
• Fixed-input logits, Top-K, and first diverging token meet gate criteria.
• Round-trip and key business regression pass.

Operational Validation

• Target engine loads on expected hardware and parallel layout.
• Memory usage, startup time, throughput, and latency show no anomalies.
• Canary traffic has independent metrics and automatic rollback.
• Source artifact remains quickly recoverable.

References

• NVIDIA Megatron Bridge — Conversion Technical Details
• NVIDIA Megatron Bridge — Parameter Mapping API
• NVIDIA Megatron Bridge — Contribute a New Model / Conversion Validation
• NVIDIA TensorRT-LLM — TensorRT-LLM Checkpoint
• Hugging Face Transformers — Loading Models and Sharded Checkpoints
• Hugging Face Safetensors