LLM Pretraining Data Deduplication in Production: Using MinHash-LSH, Semantic Deduplication, and Contamination Gates to Reduce Memorization and Benchmark Leakage
Template pages, reposted articles, and benchmark variants in LLM pretraining corpora amplify memorization, waste compute, and pollute evaluations. This article presents a complete production pipeline from exact deduplication and MinHash-LSH to semantic deduplication, benchmark isolation, and threshold rollback, covering a four-layer architecture, engineering pipeline, threshold calibration, and common pitfalls.