Background: When an Agent Moves from Demo to Production, Failure Points Multiply
Many early demos of LLM Agents resemble synchronous scripts: receive a user goal, call the model, select a tool, execute it, and return the result. This structure is clear enough in a demo environment, but once in production, several stability issues arise.
First, tasks can be long. A code-fixing Agent might need to read a repository, modify files, run tests, and iterate based on failure logs; an operations Agent might need to query data, generate a plan, wait for human confirmation, and then execute external system actions. Any timeout, process restart, container migration, or external API hiccup in the middle can cause task state to be lost.
Second, Agents execute side effects. Sending emails, submitting PRs, modifying databases, calling payment APIs, or triggering ticket workflows cannot simply be “retried on failure.” Without idempotency keys, execution logs, and state boundaries, retries can lead to duplicate submissions, duplicate charges, duplicate notifications, or state corruption.
Third, human intervention is not an exception but a part of the production flow. High-risk tool calls, low-confidence outputs, compliance text, financial operations, and pre-merge code approvals all require the Agent to pause and continue after human confirmation. The key here is not “popping up a confirmation dialog,” but ensuring the state before and after confirmation is recoverable, auditable, and replayable.
Therefore, a production-grade Agent cannot rely solely on one-shot context and in-memory variables. It should be designed as a durable workflow: every critical step has state, every external side effect has a record, every interruptible point can be recovered, and every failure has a clear retry and compensation strategy.
Note: This article is different from “Agent Memory.” Memory addresses what the Agent remembers; Durable Workflow addresses where the Agent is in execution, which side effects have occurred, and where to continue after failure.
Core Principle: Decompose Agent Execution into a Recoverable State Machine
1. Replace Implicit Call Chains with a State Machine
A common pattern in ordinary Agent code is:
result = await agent.run(user_task)This hides many production issues: what step is currently being executed? Which tools have already been called? Which tool results can be reused? How to recover before and after human approval? Which part should be re-run on failure?
A more robust approach is to break the task into a set of explicit nodes:
receive_task → plan → retrieve_context → draft_action
→ human_approval? → execute_tool → verify_result → finalizeEach node has inputs, outputs, state writes, and error handling. The value is that the task is no longer a black-box call but a traceable, recoverable execution path.
Temporal’s Workflow documentation describes workflows as a series of steps defined by code, emphasizing that Workflow Executions produce commands and events, and the event history is the basis for state recovery. External API calls, database queries, LLM calls, and file I/O should be placed in Activities; their results are recorded and reused during replay rather than re-executed.
LangGraph’s persistence documentation follows a similar approach: saving thread-level graph state via a checkpointer and using a store for cross-thread application data. It explicitly uses the checkpointer for conversation continuity, human-in-the-loop, time travel, and fault tolerance.
2. Checkpoints Are Not Chat History, They Are Recovery Cursors
Many teams store the entire history of Agent messages in a database and consider it “persisted.” This only solves part of the problem.
Chat history answers: what the model has seen before. Checkpoints answer: where the workflow execution is, what the state of each node is, and from which safe boundary to continue on recovery.
A production-ready checkpoint should at least include:
| Field | Description |
|---|---|
workflow_id | Business task ID, e.g., ticket, PR, approval form, or batch task ID |
thread_id / run_id | Recovery pointer for a single Agent execution instance |
current_node | Current or next node |
state_snapshot | Structured state, not just natural language messages |
tool_call_log | Tool call parameters, results, errors, idempotency keys |
approval_state | Human approval status, approver, approval time, approval comments |
version | Workflow definition version, prompt version, tool version |
OpenAI Agents SDK’s Sessions documentation provides a session history management mechanism, supporting implementations like SQLite, Redis, SQLAlchemy, MongoDB, Dapr, and EncryptedSession. This mechanism is suitable for maintaining Agent conversation continuity, but for long-task workflows, additional records for node state, approval state, side effect logs, and workflow version are needed.
3. Interrupts Must Be Recoverable Interrupts, Not Blocking Threads
Human review nodes are most often implemented incorrectly. The wrong approach is to have the server request wait indefinitely for human confirmation, or to store the approval state only in process memory. Once the service restarts, the approval context is lost.
LangGraph’s interrupt documentation provides a more reasonable pattern: interrupt() can pause execution within a graph node, save the current graph state, and wait for external input; recovery uses the same thread ID and Command(resume=...) to continue. The thread ID is the persistence cursor; reusing it recovers the same checkpoint, while using a new value starts a new thread.
In production, treat the human review interrupt as an explicit state:
{
"workflow_id": "ticket-20260705-001",
"status": "waiting_approval",
"current_node": "approve_tool_call",
"approval": {
"type": "send_email",
"risk_level": "medium",
"payload_digest": "sha256:...",
"requested_at": "2026-07-05T03:02:02-04:00"
}
}When approval is granted, do not initiate a completely new Agent; instead, continue with the same recovery pointer. This ensures that the same plan, the same tool parameters, and the same risk context are being approved.
Engineering Implementation: A Production Skeleton for Long-Task Agents
State Design
First, divide Agent state into four layers:
| Layer | Content | Example |
|---|---|---|
| Business State | Task goal, user input, target resource, business ID, permission context | task_id, user_id, resource_uri |
| Execution State | Current node, completed nodes, failed nodes, retry count, timeout info | current_node, retry_count, route_next |
| Model State | Prompt version, model version, tool list, context summary, draft output | prompt_version, model_name, draft_output |
| Side Effect State | Tool call ID, idempotency key, external return ID, compensation action, approval record | tool_call_id, idempotency_key, compensation |
Do not compress all of this into a single natural language summary. Natural language summaries are useful for reducing tokens but are not suitable as the sole basis for recovery. The recovery basis should be primarily structured fields, with the summary serving only as auxiliary context.
Node Boundaries
Node boundaries should be defined by “can it be safely re-run on failure?” rather than by code function size.
Safely re-runnable nodes: generating plans, reading context, computing drafts, validating output, generating summaries. These nodes, even if re-executed, generally do not directly change the external world.
Nodes requiring strict control: sending messages, submitting code, writing to databases, calling payments, initiating approvals, modifying configurations. These nodes must record idempotency keys, request digests, external return results, and compensation strategies.
Retries and Timeouts
LangGraph’s fault tolerance documentation breaks failure handling into three mechanisms: retries, timeouts, and error handling. Node failures can be retried based on exception type and backoff; timeouts can limit the duration of a single node attempt; only after retries are exhausted does the error handler take over.
In production, do not just write “retry 3 times on failure.” A more reasonable approach is to layer retry strategies by node type:
| Node Type | Retry Strategy |
|---|---|
| Model generation node | Allow short retries, but control total tokens and total duration |
| Read-only tool node | Allow exponential backoff retries |
| Write-type tool node | Allow automatic retries only when idempotency keys and confirmable query interfaces exist |
| Human review node | Do not treat as a normal timeout failure; enter a waiting state and set a business SLA |
| Summary node | Can re-run on failure, but should fix input snapshots to avoid changes in summary basis after recovery |
Side Effect Idempotency
The core risk of Agent workflows is not the model answering incorrectly, but “executing incorrectly and being unable to recover.” Therefore, every side effect action must have a unique idempotency key.
A simple strategy:
idempotency_key = hash(
workflow_id + node_name + tool_name +
normalized_payload + workflow_version
)Before executing a write tool, first check the local tool_call_log:
- If the same key has already been successfully executed → directly reuse the result
- If it is pending → query the external system to confirm
- If it failed → decide on retry, compensation, or manual handling based on the error type
When the external system does not support idempotency keys, lower the automation level. Use a pattern where “a pending execution record is generated first, then consumed by a human or a reliable background task,” rather than having the LLM Agent directly write repeatedly to critical systems.
Human Review Recovery
Human review should not only review natural language explanations but also structured content:
- Before submitting a PR: display the target branch, changed files, test results, and risk summary
- Before sending an email: display recipients, subject, body summary, and attachment list
- Before modifying a database: display the SQL, estimated row impact, and rollback plan
After approval is granted, three things must be validated upon recovery:
- Whether the approved payload digest matches the current pending payload
- Whether the workflow version and tool version are still compatible
- Whether the approval is still within its validity period
If any condition is not met, the workflow should return to re-planning or re-approval, rather than continuing with the old plan.
A Simplified Implementation Structure
Below is pseudo-code not tied to a specific framework, expressing the core structure:
class AgentWorkflowState(TypedDict):
workflow_id: str
workflow_version: str
current_node: str
task: dict
plan: dict | None
draft_action: dict | None
approval: dict | None
tool_results: list[dict]
errors: list[dict]
async def run_workflow(state: AgentWorkflowState):
state = await checkpoint.load_or_init(state["workflow_id"], state)
while state["current_node"] != "finalize":
node = state["current_node"]
try:
if node == "plan":
state["plan"] = await call_model_for_plan(state["task"])
state["current_node"] = "draft_action"
elif node == "draft_action":
state["draft_action"] = await create_action_payload(state)
state["current_node"] = "approval"
elif node == "approval":
approval = await approval_store.get(state["workflow_id"])
if not approval:
await approval_store.create(
state["workflow_id"], state["draft_action"]
)
state["current_node"] = "waiting_approval"
await checkpoint.save(state)
return {"status": "waiting_approval"}
validate_approval(approval, state["draft_action"])
state["approval"] = approval
state["current_node"] = "execute_tool"
elif node == "execute_tool":
key = build_idempotency_key(state)
result = await tool_executor.run_once(
key, state["draft_action"]
)
state["tool_results"].append(result)
state["current_node"] = "verify_result"
elif node == "verify_result":
await verify_result(state)
state["current_node"] = "finalize"
await checkpoint.save(state)
except RetryableError as e:
await retry_policy.wait_and_retry(node, e)
except Exception as e:
state["errors"].append(serialize_error(e))
await checkpoint.save(state)
raise
return {"status": "done", "state": state}The focus of this code is not the framework API, but execution discipline: save state after each node, manage idempotency for side effects via run_once, write state to checkpoints during approval pauses, and continue from the same workflow ID on recovery.
Applicable Scenarios
| Scenario | Typical Characteristics | Key Requirements |
|---|---|---|
| Long-running code Agent | Multiple rounds of model calls, command execution, test fixing, human confirmation, and PR submission | Recovery after interruption, avoid duplicate operations |
| Enterprise process Agent | Ticket processing, contract review, data report generation, operational action execution | Business state management, human review nodes, side effect control |
| Semi-automated operations Agent | Diagnose alerts, generate fix plans, wait for SRE approval, execute security scripts | Approval boundaries, audit trail, safe recovery |
| High-value content production Agent | Auto-write articles, generate images, send emails, publish to CMS | Prevent duplicate sending, recovery after interruption, publish state tracking |
Common Misconceptions
Misconception 1: Saving Chat History Equals Recoverability
Chat history cannot replace workflow state. Recovery requires knowing the execution node, tool results, approval status, and whether side effects have occurred. Chat history can serve as model context but should not be the sole source of truth.
Misconception 2: Automatically Retry All Failures
Retry strategies differ for model calls, read-only queries, and idempotent writes. For irreversible side effects, automatic retries must be built on idempotency keys, confirmation queries, and compensation mechanisms.
Misconception 3: Human Review Only Needs a Confirm Button
Human review nodes must save a structured snapshot and digest hash of the content being reviewed. Otherwise, after approval, the actual executed content may have been silently altered by subsequent model calls or code changes.
Misconception 4: Workflow Versions Can Be Changed Anytime
Long tasks may span hours or even days. After a new version is released, how do old tasks continue? Should they use the old version or migrate to the new one? Does migration require re-approval? These require a version strategy, not letting old tasks run new code directly.
Misconception 5: The More Complete the State, the Better
Checkpoints are not unlimited log warehouses. Overly large states increase read/write costs, recovery latency, and privacy risks. In practice, retain structured core state and store large files, long logs, and model context summaries separately.
Production Readiness Checklist
| Dimension | Check Item |
|---|---|
| State & Recovery | Is state written after each critical node? Can recovery continue from the same workflow ID after a service restart? Will successfully executed tool calls be re-executed on recovery? |
| Side Effect Safety | Do all write-type tools have idempotency keys? Is there a tool call log? Can the external system be queried for confirmation when it succeeds but the local system fails? |
| Human Review Process | Is the review content displayed in a structured format? Is there a digest check for the review payload? Does the review prevent continued execution after expiration? Does rejection lead to a clear final state or re-planning? |
| Retries & Timeouts | Do different nodes have independent retry policies? Is there a distinction between run timeout and idle timeout? Is there an error handler or manual processing entry after retries are exhausted? |
| Version Governance | Are workflow definitions, prompts, tool schemas, and model configurations versioned? Is there a compatibility strategy for old tasks when a new version is released? |
| Observability | Can the complete execution trace be queried by workflow_id? Are node duration, failure reasons, retry counts, human wait times, and external system return IDs recorded? |
| Data Governance | Is sensitive information stored in checkpoints? Are there retention periods, encryption, masking, and deletion policies? |
Summary
Productionizing an LLM Agent is not just about making the model smarter; it’s also about making the execution process more controllable. The goal of a durable workflow is to decompose an opaque Agent run into a recoverable state machine: each node has a checkpoint, each side effect has an idempotency key, each human review has a recovery pointer, and each failure has a clear path for retry, compensation, or manual handling.
When Agents start handling code, tickets, emails, approvals, operations, and business data, this mechanism becomes more important than simply optimizing prompts. Because real production incidents are often not about the model saying one less sentence, but about the system not knowing where it is in execution, or whether it can safely continue to the next step.