Tag: Performance Engineering

4 articles

• LLM Tokenizer Preprocessing Pool in Production: Avoiding GPU Starvation with Async Tokenizer Pools, Length-Aware Queues, and CPU Affinity

  The bottleneck in LLM serving isn't always the GPU. This article explains how to decouple the tokenizer preprocessing pool using async batch encoding, length-aware queues, CPU affinity, and backpressure metrics to prevent long prompts from blocking requests and keep the GPU fed with executable batches.

  07/22/2026 LLM Serving / Tokenizer / Performance Engineering / CPU Optimization / vLLM

• LLM GPU Energy Efficiency in Production: Phase-Aware Clock Locking, Joules/Token, and Thermal Throttling Guardrails

  Setting a Power Cap alone often ignores the load differences between Prefill and Decode. This article explains how to use phase-aware clock locking, per-token energy metrics, and thermal throttling guardrails to manage inference energy efficiency without breaking latency SLOs.

  07/13/2026 LLM Serving / GPU Energy Efficiency / NVIDIA DCGM / Clock Lock / Performance Engineering

• Continuous Batching in Production: Keeping LLM Services GPU-Saturated Under High Concurrency

  A systematic guide to continuous batching for LLM serving: core principles, scheduling strategies, KV Cache constraints, engineering deployment steps, and monitoring metrics. Covers the full path from proof-of-concept to production, helping teams boost throughput while controlling tail latency in high-concurrency scenarios.

  07/01/2026 LLM Serving / Continuous Batching / vLLM / KV Cache / Performance Engineering / Inference Optimization / GPU Scheduling

• Speculative Decoding in Production: Reducing LLM Latency with Draft Models

  A systematic guide to Speculative Decoding for LLM inference: principles, engineering trade-offs, model selection, monitoring metrics, and common pitfalls. Learn when and how to deploy draft models for latency reduction.

  06/30/2026 LLM / Inference Optimization / Speculative Decoding / LLMOps / Performance Engineering