Does Sequence Packing cause different samples to see each other's content?

Whether cross-sample contamination occurs depends on the implementation. Simply concatenating input_ids with EOS tokens is insufficient. The attention kernel must recognize each sample's boundaries and synchronously handle labels and position_ids.

Should I choose between Length Bucket and Sequence Packing?

No. Length Bucket groups samples of similar length, and Packing packs them into a fixed token budget. They can be combined, but you need to recalculate the effective token count per step.

Can I just look at samples per second after enabling packing?

No. You should also monitor effective tokens per second, padding ratio, training loss, validation quality, gradient stability, and token count per optimizer step.

Why might validation loss change after enabling Packing?

Common causes include unisolated cross-sample attention, unmasked boundary labels, different position ID handling, varying effective token counts per step, changed sample order, or numerical differences in attention kernels. Use a fixed small sample for token-level comparison before adjusting the learning rate.

LLM SFT Sequence Packing in Production: Boosting Throughput with Sample Boundaries, Loss Mask, and Length Buckets

Background: Much SFT Compute is Wasted on Padding

Supervised fine-tuning (SFT) data often has wildly varying lengths: one conversation might be a few dozen tokens for a classification answer, while another could be thousands of tokens for code generation or long-form analysis. Traditional batch training requires padding all samples in a batch to the length of the longest sequence. The numerous [PAD] tokens after short samples still consume GPU memory, trigger partial kernel work, and lengthen each training iteration.

Before optimizing, quantify the problem with a simple metric:

Effective Token Utilization = Non-Padding Tokens ÷ Total Tokens Computed

If a batch has only a few long samples and many short ones padded to the same length, samples/s may look normal, but tokens/s could be poor. Sequence Packing aims to pack multiple short samples into a fixed-length training block, letting the GPU process real training tokens instead of blank placeholders.

But Packing is not just concatenating arrays. A truly safe production implementation must guard three boundaries:

Attention Boundary: Sample B must not read sample A’s context.
Loss Boundary: Prompt, System, User, and cross-sample connection positions must not be incorrectly included in the loss.
Position Boundary: Position IDs must match the model and attention kernel’s expectations, avoiding unverified distribution shifts from concatenation.

Core Principle: Upgrading “Concatenation” to Boundary-Aware Packing

Differences Between Padding, Length Bucket, and Packing

Strategy	Principle	Pros	Cons
Padding	Pad short samples to the longest sequence in the batch	Simple implementation, best compatibility	Lots of wasted computation, low memory utilization
Length Bucket	Group samples by length first, so samples in the same batch are closer in length	Reduces outer padding, compatible with regular tensor shapes	Still has padding waste, bucket tuning needed
Sequence Packing	Pack multiple samples into one token container	Nearly eliminates padding, highest token utilization	Complex implementation, requires attention backend support

Length Bucket retains the regular tensor shape of “one sample per row” and has the best compatibility. Sequence Packing packs multiple samples into one token container, with the total length close to max_length, carrying each sample’s start and end positions. For implementations supporting Variable-Length Attention, the kernel can use the cumulative length array cu_seqlens to compute each sample directly, without constructing a large full attention mask.

They are not mutually exclusive. A common best practice is to first use Length Bucket to limit length variation, then perform Packing within the bucket.

All Three Boundaries Must Be Designed Together

Assume three samples are concatenated into one token stream:

[A0 A1 A2 EOS] [B0 B1 EOS] [C0 C1 C2 C3 EOS]

Simply inserting EOS does not guarantee attention isolation. A standard causal mask still allows later tokens to see all previous positions. A production implementation needs block-diagonal attention, cu_seqlens, sequence IDs, or equivalent metadata supported by the backend to explicitly declare the three segments as independent sequences.

The loss side also requires careful handling. Typical rules in SFT include:

Set labels for System, User, tool results, and other input tokens to -100 (ignore loss);
Compute loss only on Assistant or Completion parts;
Based on the data template and label shift logic, exclude the first token of each sample and cross-sample connection positions to prevent the previous sample from being forced to “predict the start of the next sample”;
Whether EOS participates in the loss must be solidified as a data contract, not left to different collators to decide.

Position IDs are usually reset to zero at each sample boundary, but this is not an unconditional rule. Whether to reset and how to pass it to the FlashAttention Variable-Length interface must be consistent with the model architecture, RoPE configuration, and training framework.

Engineering Implementation: From Length Profiling to Release Gates

Step 1: Quantify Padding Waste First

Don’t just turn on packing=True. First, compute these metrics offline:

P50, P90, P95, P99 sample lengths;
Proportion of samples exceeding max_length;
Effective token utilization per batch;
Ratio of prompt tokens to assistant tokens;
Length distributions across different tasks, languages, and data sources.

It’s recommended to compute per data domain. Mixing code tasks, customer service Q&A, and long-form summaries in one global histogram can mask local extreme distributions.

Step 2: Choose a Packing Strategy

Hugging Face TRL currently offers several packing approaches:

Strategy	Behavior	Use Case
BFD (Best-Fit Decreasing)	Sort by length, then place into the best-fitting remaining space	Default starting point, does not actively cut samples
BFD Split	Allows splitting long samples to retain more tokens	Only if task semantics allow splitting and the contract is verified
Wrapped	Treat as a continuous token stream, cut into fixed-length chunks	High space utilization, but easily cuts samples and mixes unrelated content

For most enterprise SFT, the recommended priority order is: Fix truncation/filtering rules first → Use Length Bucket to reduce dispersion → Then use BFD Packing without splitting samples. Do not go directly to the Wrapped strategy just for higher throughput.

Step 3: Define a Unified Pack Data Structure

A production collator should at least output or allow tracing of the following fields:

packed_batch = {
    "input_ids": ...,     # Concatenated tokens
    "labels": ...,        # Loss mask already applied for Prompt/Assistant
    "position_ids": ...,  # Position IDs for each sample
    "cu_seqlens": ...,    # [0, len_a, len_a + len_b, ...]
    "max_seqlen": ...,    # Longest independent sample in the pack
    "sample_ids": ...,    # For traceability and auditing
}

sample_ids may not enter the model computation graph, but should be retained in training logs or side-channel metadata. When loss spikes, NaN, or specific sample contamination occurs, you must be able to trace back from the pack to the original sample and template version.

Step 4: Treat Loss Mask as a Data Contract

TRL’s SFT Trainer supports Packing and Assistant-Only Loss:

from trl import SFTConfig

config = SFTConfig(
    max_length=4096,
    packing=True,
    packing_strategy="bfd",
    assistant_only_loss=True,
)

But assistant_only_loss=True is not a universal switch. It relies on the Chat Template providing a generation mask corresponding to the Assistant interval. If the template doesn’t support it, labels must be explicitly generated during preprocessing.

Pre-deployment assertions:

def validate_pack(input_ids, labels, position_ids, cu_seqlens):
    assert cu_seqlens[0] == 0
    assert cu_seqlens[-1] == len(input_ids)
    assert all(b > a for a, b in zip(cu_seqlens, cu_seqlens[1:]))
    for start, end in zip(cu_seqlens[:-1], cu_seqlens[1:]):
        assert position_ids[start] == 0
        assert end > start
        # Verify prompt region labels == -100 per project template
        # Verify at least one trainable label exists in the assistant region

A stricter approach is to perform unpacked comparison replay: run the same small batch of samples with both the unpacked baseline and the packed implementation, comparing:

Number of trainable labels;
Token sequence for each sample;
Per-sample loss;
Gradient norms;
If the backend claims equivalence, compare numerical errors in logits or gradients.

Step 5: Verify the Attention Backend Actually Isolates Samples

Padding-Free Training typically relies on FlashAttention 2/3 or equivalent Variable-Length Attention. Don’t just check if the config option starts successfully; verify the actual execution path:

Is the Variable-Length Kernel being called?
Does cu_seqlens cover all tokens?
Is there unexpected attention between different samples in the pack?
Is it still safe when falling back to regular attention?
Is the path compatible with mixed precision, Gradient Checkpointing, FSDP/DeepSpeed?

The NVIDIA Megatron Bridge documentation on Packed Sequences also emphasizes that support scope depends on the specific model, recipe, and training features; some offline packed SFT configurations may require specific micro-batch constraints. Don’t interpret “framework supports Packing” as “all models and parallelism combinations are verified.”

Step 6: Redefine Training Steps and Scheduling Metrics

After Packing, the number of samples and tokens in a batch can change. If you still understand training scale by the original batch_size × gradient_accumulation_steps, learning rate scheduling, logging, and cost accounting may all be off.

It’s recommended to elevate the following metrics to first-class monitoring objects:

Effective training tokens per optimizer step;
Effective training tokens per second;
GPU time per million effective tokens;
Proportion of tokens entering training per data domain;
Ratio of padded, truncated, and discarded tokens;
Pack fill rate and its distribution.

For multi-task data, sample by token budget rather than sample count, otherwise short-sample tasks may get unexpected weight changes due to Packing.

Applicable Scenarios

Sequence Packing is more suitable for:

✅ SFT, PEFT, or multi-task training with significant sample length variation
✅ Many samples much shorter than max_length
✅ Attention backend supports reliable Variable-Length or boundary-aware computation
✅ Team can maintain sample-to-pack mapping and perform unpacked comparison replay
✅ Training cost is primarily driven by padding waste

It is not a substitute for long-context training. When a single sample exceeds the context window, you still need truncation, sample splitting, Context Parallelism, or model-level long-context solutions. Packing solves “how multiple short samples share one compute container,” not “how one very long sample fits into the model.”

Common Misconceptions

Misconception 1: EOS Prevents Cross-Sample Attention

EOS is a semantic separator, not a physical attention mask. If the backend still uses a standard causal mask, later samples can still access earlier tokens.

Misconception 2: Only Verify Training Runs

Cross-sample attention and incorrect loss masks often don’t cause immediate errors; loss may even drop faster. You must perform unpacked comparisons, boundary assertions, and sample-level replay.

Misconception 3: Only Pursue Maximum Pack Fill Rate

The Wrapped strategy may achieve higher utilization but cuts samples or changes training semantics. The production goal should be the optimal point under effective token throughput and quality constraints, not a single fill rate.

Misconception 4: Treat samples/s as the Primary Metric

Packing changes the number of samples in a step. Prioritize effective tokens/s, tokens per step, and validation set quality.

Misconception 5: Ignore Data Mixing Weights

When sampling by sample, the token contribution from tasks of different lengths varies greatly. Packing can amplify hidden length biases. Check token proportions by task, language, and data source.