Tag: LLM Engineering

5 articles

• LLM Batch API Offline Inference Production Guide: Reducing Costs with JSONL Queues, custom_id, and Result Backfill

  This article explains how to transform non-real-time LLM tasks into a recoverable Batch API offline pipeline, covering JSONL queues, custom_id, status polling, error backfill, result reconciliation, cost budgeting, and a go-live checklist to help teams reduce costs and improve engineering reliability for large-scale LLM tasks.

  07/07/2026 LLM Batch API / Offline Inference / Cost Optimization / LLM Engineering

• Voice Agent in Production: Stabilizing Speech LLM Experiences with Streaming ASR, VAD, and Segmented Context

  A deep dive into the real-time pipeline for speech LLM applications from an engineering perspective, covering how to coordinate streaming ASR, VAD, segmented context, LLM inference, and TTS to reduce end-to-end latency and maintain semantic boundaries in long conversations.

  07/02/2026 Voice Agent / Streaming ASR / VAD / LLM Engineering / Realtime AI / Speech LLM

• Structured Output in Production: Using Constrained Decoding to Make LLMs Return Reliable JSON

  A practical guide to structured output and constrained decoding for LLMs. Learn how to use JSON Schema, strict mode, post-hoc validation, retry strategies, and production monitoring to ensure stable, parseable structured results for information extraction, agent state management, and automated workflows, reducing online risks at the source.

  06/30/2026 LLM / Structured Output / Constrained Decoding / JSON Schema / LLM Engineering

• Structured Outputs in Production: Stabilizing JSON Schema Generation with Constrained Decoding

  A systematic guide to Structured Outputs and Constrained Decoding: core principles, schema design, server integration, performance, error handling, and common pitfalls for reliable LLM JSON generation.

  06/30/2026 LLM Engineering / Structured Outputs / Constrained Decoding / JSON Schema

• PagedAttention in Action: Why KV Cache Scheduling Matters More Than Compute for LLM Serving

  A deep dive into KV Cache memory fragmentation, PagedAttention, continuous batching, and prefix caching — and how they boost throughput and reduce latency in LLM inference.

  06/29/2026 LLM Inference / PagedAttention / KV Cache / vLLM / LLM Engineering