Tag: Idempotency

2 articles

• LLM Batch Inference Production Practices: Avoiding Duplicate Charges and Missing Results with Manifest, Idempotent Record IDs, and Shard Retry

  A systematic guide to building a recoverable, auditable batch inference pipeline for asynchronous large-scale LLM calls using job manifests, deterministic record IDs, per-record reconciliation, and shard-based retries to prevent duplicate billing and missing results.

  07/28/2026 LLM batch inference / idempotency / job orchestration / reliability engineering / cost governance / shard retry

• Tool Result Caching in Production: Reducing Agent External Call Costs with Cache Hit Rates

  A systematic guide to caching tool call results in LLM agents, covering cache key design, TTL, idempotency, invalidation, observability, and production checklists to reduce external API costs and tail latency while maintaining accuracy.

  07/02/2026 LLM Agent / Tool Calling / Caching / Cost Optimization / Production / Cache Key / TTL / Idempotency