Engineering the future of large models and AI agents

Latest Articles

• LLM Pretraining Data Mixture in Production: Controlling Capability Bias with Mixture Manifest, Token Budget, and Domain Loss Feedback

  A systematic guide to engineering data mixtures for LLM pretraining: using immutable Mixture Manifests to freeze domain weights and corpus versions, Token Budgets for precise accounting, and proxy model search, domain loss feedback, starvation protection, and replay gates to continuously calibrate capability structure. Turn mixture ratios from probabilistic parameters into auditable, reproducible training contracts.

  07/29/2026 LLM Pretraining / Data Mixture / DoReMi / RegMix / Data Governance / Token Budget / Mixture Manifest

• LLM μP Hyperparameter Cross-Scale Transfer in Production: Reducing Large Model Tuning Costs with Base Shape Fingerprints, Coord Checks, and Proxy Sweeps

  A systematic breakdown of μP hyperparameter cross-scale transfer principles and engineering boundaries, covering how to solidify Base Shapes, run Coord Checks, organize proxy model sweeps with regression gates, and avoid misapplying width transfer to training duration, batch size, or architectural changes.

  07/28/2026 LLM training / μP / hyperparameter transfer / pretraining engineering / scaling

• LLM Knowledge Distillation in Production: Controlling Student Model Degradation with Teacher Logprob Contracts, On-Policy Replay, and Top-K Tail Buckets

  A practical engineering guide to productionizing LLM knowledge distillation, covering Teacher Logprob contracts, temperature and divergence selection, On-policy replay, Top-K tail probability buckets, Tokenizer alignment, and student capability gating to prevent silent degradation and deployment failures.

  07/28/2026 LLM / Knowledge Distillation / Model Compression / On-policy Distillation / MLOps

• LLM Batch Inference Production Practices: Avoiding Duplicate Charges and Missing Results with Manifest, Idempotent Record IDs, and Shard Retry

  A systematic guide to building a recoverable, auditable batch inference pipeline for asynchronous large-scale LLM calls using job manifests, deterministic record IDs, per-record reconciliation, and shard-based retries to prevent duplicate billing and missing results.

  07/28/2026 LLM batch inference / idempotency / job orchestration / reliability engineering / cost governance / shard retry

• LLM Model Format Conversion in Production: Avoiding Silent Degradation with Tensor Mapping, QKV Reordering, and Layer-Wise Validation

  A systematic guide to parameter mapping, QKV reordering, shard indexing, layer-wise numerical validation, and rollback gates when converting LLM weights between Hugging Face, Megatron, and TensorRT-LLM. Build a repeatable, auditable release pipeline to prevent silent degradation, output drift, and parallel slicing errors.

  07/22/2026 LLM Engineering / Model Conversion / Megatron / TensorRT-LLM / Safetensors

• LLM Tokenizer Preprocessing Pool in Production: Avoiding GPU Starvation with Async Tokenizer Pools, Length-Aware Queues, and CPU Affinity

  The bottleneck in LLM serving isn't always the GPU. This article explains how to decouple the tokenizer preprocessing pool using async batch encoding, length-aware queues, CPU affinity, and backpressure metrics to prevent long prompts from blocking requests and keep the GPU fed with executable batches.

  07/22/2026 LLM Serving / Tokenizer / Performance Engineering / CPU Optimization / vLLM