LLM Pretraining Data Mixture in Production: Controlling Capability Bias with Mixture Manifest, Token Budget, and Domain Loss Feedback
A systematic guide to engineering data mixtures for LLM pretraining: using immutable Mixture Manifests to freeze domain weights and corpus versions, Token Budgets for precise accounting, and proxy model search, domain loss feedback, starvation protection, and replay gates to continuously calibrate capability structure. Turn mixture ratios from probabilistic parameters into auditable, reproducible training contracts.