Tag: Engineering Practice

3 articles

• Speculative Decoding in Production: Reducing LLM Decode Latency with Draft-Verify

  A systematic guide to Speculative Decoding's Draft-Verify mechanism, covering vLLM and TensorRT-LLM integration, metric tuning, applicability boundaries, and production deployment checks to reduce decode latency and avoid overhead from low acceptance rates.

  07/01/2026 LLM / Inference Optimization / Speculative Decoding / LLM Serving / Engineering Practice

• Prefill-Decode Disaggregation in Production: Separating Prefill and Decode to Reduce TTFT/TPOT Conflicts

  A systematic guide to disaggregating LLM prefill and decode phases. Covers the root cause of TTFT vs TPOT conflicts in long-context scenarios, KV cache transfer, dynamic routing, resource allocation, monitoring metrics, and a production launch checklist.

  07/01/2026 LLM / Inference Acceleration / LLM Serving / KV Cache / Engineering Practice

• LLM-as-a-Judge in Production: How to Calibrate Automated Evaluation Without Treating the Model as an Oracle

  A systematic guide to deploying LLM-as-a-Judge for evaluating LLM applications in production, covering dataset construction, rubric design, bias calibration, metric monitoring, and human-in-the-loop review.

  07/01/2026 LLM / LLM Evaluation / LLM-as-a-Judge / Engineering Practice / LLMOps