Background: Duplicate Data Is More Than a Storage Waste
Duplicate content in pretraining corpora typically comes from web mirrors, reposted news, templated product pages, site farms, code repository forks, repeated synthetic data generation, and translated or rewritten versions of the same benchmark questions.
These duplicates don’t just increase disk usage. They alter the effective weight of training samples, causing a small set of texts to be seen repeatedly by the model, leading to four production issues:
- Training compute is consumed by repeated content, reducing effective data density.
- Risk of memorization and verbatim reproduction increases, especially for high-frequency long texts and sensitive fragments.
- Validation and training sets leak, inflating offline metrics.
- Data versioning becomes unexplainable, and teams cannot answer why a particular sample was kept or removed.
Lee et al. found a large number of near-duplicate documents and long repeated fragments in multiple language model datasets, and observed a significant decrease in the frequency of verbatim reproduction of training text after deduplication. This conclusion cannot be directly extrapolated to all models and corpora, but it is sufficient to show: Deduplication is a quality control measure for training data, not a simple storage optimization.
Core Principles: Breaking Down “Duplication” into Four Levels
A production system should not use a single algorithm and a single threshold to handle all duplicates. A more robust approach is to break the problem into four layers.
Layer 1: Normalized Exact Document Deduplication
First, apply auditable lightweight normalization to the text, such as unifying line endings, Unicode normalization forms, and deterministic whitespace handling, then compute a content hash.
This layer is suitable for handling:
- Identical web snapshots;
- Duplicate imported files;
- Multiple concatenations of the same corpus;
- Copies produced by rerunning collection tasks.
⚠️ Normalization rules themselves must be versioned. If you directly remove punctuation, digits, case, or HTML structure, you may incorrectly merge code, formulas, tables, and entity information that are actually different.
Layer 2: Exact Long Substring Deduplication
Two documents may be entirely different yet share very long repeated passages. Common scenarios include site headers and footers, legal disclaimers, reposted body text, and forum quotes.
You can use suffix arrays, rolling hashes, or chunk-based fingerprints to find long repeated fragments. The work of Lee et al. combined exact long substring matching with near-duplicate document matching, showing that they address different problems.
In production, you should not just record “document deleted.” You should also record:
| Field | Description |
|---|---|
| Hit fragment range | Start and end positions of the repeated content in the document |
| Fragment length | Number of tokens or characters |
| Source document | Which document the repeated fragment came from |
| Retained document | Which document is kept after deduplication |
| Rule version and batch | Version number and run batch of the deduplication rule |
Layer 3: MinHash-LSH Near-Duplicate Document Clustering
MinHash uses a set of hash signatures to approximate the Jaccard similarity of two documents’ shingle sets; LSH (Locality-Sensitive Hashing) places potentially similar documents into the same candidate buckets, avoiding full pairwise comparison.
The typical workflow is:
- Tokenize the document and construct n-gram shingles;
- Generate MinHash signatures;
- Use LSH to produce candidate pairs;
- Perform more precise Jaccard, edit similarity, or containment checks on candidate pairs;
- Build a duplicate relationship graph and compute connected components;
- Keep only one deterministic representative document per duplicate cluster.
⚠️ LSH is only a candidate recall mechanism and should not directly make the final deletion decision. Otherwise, hash collisions and threshold boundaries can turn false positives into irreversible data loss.
Layer 4: Semantic Deduplication and Benchmark Contamination Checks
String-level deduplication cannot reliably identify translations, paraphrases, variable substitutions, or answer order shuffling. Research like SemDeDup uses pretrained model embeddings to find semantic duplicates, showing that semantic redundancy can persist even when text surfaces differ significantly.
However, semantic deduplication also carries higher risk. Two articles with similar content may come from different languages, different viewpoints, or different professional contexts. In production, a layered approach is more suitable:
- Perform secondary subdivision on oversized MinHash clusters;
- Check for repeated generation in synthetic data;
- Conduct manual sampling for high-value sources;
- Establish independent contamination scans for benchmark questions, answers, translations, and rewritten versions.
⚠️ Benchmark contamination should not be mixed with general corpus deduplication. It should be a separate gate, because its goal is not to compress the corpus, but to protect the credibility of evaluations.
Engineering Implementation: Building a Replayable Deduplication Pipeline
1. First, Create Immutable Data Snapshots and Stable IDs
Each raw record should retain at least:
| Field | Description |
|---|---|
document_id | Stable unique identifier |
source_id | Data source identifier |
| Original URI / object storage path | Traceable storage path |
| Crawl timestamp | Data acquisition timestamp |
| Original content hash | Hash of unnormalized content |
| Language, document type, and license info | Metadata |
| Data batch version | Batch identifier |
The deduplication layer only generates retention, clustering, and exclusion lists; it does not directly overwrite the original objects. This allows replay when thresholds change or false deletions occur.
2. Treat Normalization Rules as a Data Contract
Below is an example configuration. Thresholds must be calibrated based on real corpus sampling; do not copy them as unified defaults.
dedup_pipeline:
version: "2026-07-14-v1"
normalization:
unicode_form: "NFC"
normalize_line_endings: true
collapse_repeated_spaces: true
remove_boilerplate: false
exact_document:
hash: "sha256"
exact_substring:
enabled: true
minimum_match_tokens: "${CALIBRATED_MIN_MATCH_TOKENS}"
near_duplicate:
shingle_size: 5
num_permutations: "${CALIBRATED_NUM_PERMUTATIONS}"
lsh_threshold: "${CALIBRATED_LSH_THRESHOLD}"
secondary_verifier: "jaccard_and_edit_similarity"
semantic_duplicate:
enabled_for:
- synthetic_data
- oversized_duplicate_clusters
- benchmark_decontamination
embedding_model_version: "${PINNED_MODEL_VERSION}"
similarity_threshold: "${CALIBRATED_SEMANTIC_THRESHOLD}"3. Representative Documents Must Be Deterministically Selected
Do not simply keep the “first arrived” record in a duplicate cluster, as parallel task reruns may produce different results. Use a stable sorting rule:
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class Document:
document_id: str
source_priority: int
quality_score: float
license_ok: bool
captured_at: datetime
token_count: int
def representative_key(doc: Document) -> tuple:
return (
0 if doc.license_ok else 1,
-doc.source_priority,
-doc.quality_score,
-doc.token_count,
doc.captured_at,
doc.document_id,
)
def choose_representative(cluster: list[Document]) -> Document:
if not cluster:
raise ValueError("duplicate cluster must not be empty")
return min(cluster, key=representative_key)The sorting criteria should be written into the data dictionary and saved with the deduplication results. If rules are upgraded, a new data version must be generated, not an in-place modification of old results.
4. Place Contamination Gates Both Before and After Data Splitting
It is recommended to perform two checks:
- Before splitting: Use the retained evaluation set fingerprints to scan the full raw corpus and remove clearly hit training candidates.
- After splitting: Recheck for exact, near-duplicate, and semantic overlaps between training, validation, and test sets to prevent cluster splitting or subsequent incremental data from reintroducing contamination.
For public benchmarks, maintain at least three types of fingerprints:
| Fingerprint Type | Description |
|---|---|
| Normalized text and n-gram fingerprints | String-level exact matching |
| Combined fingerprints after splitting questions and answers | Prevents concatenation from bypassing detection |
| Semantic representations of translations, rewrites, and code variable substitutions | Covers semantic-level leakage |
Work by Yang et al. shows that simple rewrites or translations can bypass string-based contamination detection. Therefore, semantic scanning should be a supplementary method for high-risk benchmarks, not a complete replacement for exact matching.
5. Don’t Recompute Everything from Scratch for Incremental Data
In scenarios like continuous pretraining and periodic crawling, maintain a versioned signature index:
- New documents are first compared against the historical retention set;
- New batches are then clustered internally;
- Only affected connected components are recomputed;
- Both deletions and restorations are written to a change log;
- Periodic full audits are performed to prevent index drift.
Avoid representative document oscillation, where “A is kept today, B is kept tomorrow.” Historical representative documents should remain stable as long as their quality has not significantly degraded.
How to Calibrate Thresholds
Thresholds should not be copied directly from papers or open-source configurations. A more reliable method is to build labeled samples per language, source, and document type.
Each threshold version should evaluate at least the following metrics:
| Evaluation Dimension | Specific Metric |
|---|---|
| Deduplication quality | Precision and Recall of duplicate pair identification |
| Data impact | Proportion of tokens removed |
| Source balance | Retention ratio per source and language |
| Cluster distribution | Number of oversized duplicate clusters and maximum cluster size |
| Contamination control | Cross-hit rate between training and validation sets |
| Benchmark safety | Exact, near-duplicate, and semantic hit rates on benchmarks |
| Model effect | Training loss curve before and after deduplication |
| Regression validation | Downstream capabilities, long-tail domains, and multilingual regression |
Before changing thresholds, run a Dry Run to output “what will be deleted,” and obtain approval from the data owner. Set independent strategies for code, mathematics, legal text, and minority languages, because similar templates and fixed expressions do not necessarily mean low-value duplication.
Applicable Scenarios
This approach is suitable for:
- General large model pretraining corpora;
- Continuous pretraining of domain-specific models;
- Large-scale SFT or synthetic data construction;
- Repository and fork governance for code models;
- Multilingual web corpora;
- Model development workflows requiring credible benchmark evaluations.
For small-scale, highly controlled manual datasets, exact hashing and manual review may be sufficient, and there is no need to introduce expensive full-scale semantic deduplication.
Common Pitfalls
Pitfall 1: Higher Deduplication Rate Means Cleaner Data
The deduplication rate is a result metric, not an optimization target. Over-deduplication may remove legitimate duplicates, minority language samples, and important domain expressions.
Pitfall 2: One Threshold Fits All Sources
The similarity distributions of news reposts, forum conversations, source code, patents, and product pages are completely different. A global threshold typically causes both false positives and false negatives.
Pitfall 3: Semantic Similarity Means Only One Copy Should Be Kept
Semantic similarity does not mean data equivalence. Viewpoints, fact timeliness, licenses, language, and professional context may still differ.
Pitfall 4: Only Check Contamination After Training
Post-hoc checks can only explain risk; they cannot recover the training compute already consumed or fully restore the credibility of contaminated evaluations.
Pitfall 5: Deduplication Results Can Directly Overwrite Raw Data
Without original snapshots, cluster evidence, and rule versions, false deletions cannot be recovered, and thresholds cannot be replayed.
Go-Live Checklist
- Raw corpus snapshots are immutable; document IDs are stable.
- Normalization, tokenizer, shingle, and hash rules are all versioned.
- LSH candidates undergo secondary similarity verification.
- Representative document selection in duplicate clusters is deterministic.
- Deletion lists include hit evidence, source, and rule version.
- Bidirectional contamination scanning has been performed between training, validation, and test sets.
- Public benchmarks include checks for rewrites and translation risks.
- Thresholds are evaluated by sampling per language, source, and document type.
- Retention ratios for long-tail domains and minority languages are monitored.
- Supports Dry Run, rollback, and historical version replay.
- Small-scale training and capability regression are performed after deduplication.
- Incremental signature indexes have integrity checks and periodic full audits.
References
- Deduplicating Training Data Makes Language Models Better — arXiv:2107.06499
- SemDeDup: Data-efficient learning at web-scale through semantic deduplication — arXiv:2303.09540
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples — arXiv:2311.04850
- Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research — arXiv:2402.00159
- D4: Improving LLM Pretraining via Document De-Duplication and Diversification — arXiv:2308.12284