Agent Durable Execution in Production: Preventing Duplicate Long-Running Tasks with Checkpoints, Idempotent Side Effects, and Replay Contracts

Background: The Most Dangerous Agent Failure Isn’t “Failure” — It’s “Duplicate Success After Recovery”

A single failed chat turn is usually easy to handle: return an error and let the user retry. But production-grade agents are often state machines that run for minutes or even hours:

1. Call an LLM to plan the task;
2. Query a database;
3. Create a ticket;
4. Wait for human approval;
5. Call a payment or order API;
6. Send an email;
7. Summarize the results.

If the worker crashes after step 5, the system must know: which of the first four steps have completed, whether step 5 actually landed in the external system, and whether step 6 should proceed.

At this point, saving conversation history alone is not enough. The goal of Durable Execution isn’t to “remember more context” — it’s to create a recoverable consistency boundary between execution progress, non-deterministic results, and external side effects.

LangGraph, Temporal, and Microsoft Agent Framework have different APIs, but they all expose three problems that must be governed separately:

• Checkpoint Boundary: where to resume from;
• Side-effect Boundary: which external actions might be duplicated;
• Replay Contract: which code can be re-executed during recovery, and which results must reuse historical records.

Understanding these three boundaries matters more than simply “adding a Redis state table to the agent.”

Core Principle 1: Checkpoints Save Execution State, Not Exactly-Once Guarantees

Checkpoints easily create the illusion that since state is saved, execution won’t be duplicated after recovery. In reality, a checkpoint can only answer “what does the system know at recovery time” — it cannot automatically answer “what has already happened in the outside world.”

LangGraph’s checkpoints save graph state at execution boundaries, and the official docs explicitly warn that nodes may re-execute from the beginning after an interrupt or resume. Side effects that occurred before the interrupt must therefore be idempotent or be extracted into separate Tasks/Nodes. Microsoft Agent Framework similarly creates checkpoints after each superstep, saving executor state, pending messages, pending requests/responses, and shared state. This allows workflow recovery, but it does not mean any arbitrary external write operation naturally gets exactly-once semantics.

For production design, you should therefore split state into two categories:

1. Replayable State (suitable for checkpoints)

• Current step;
• Planner output;
• Tool results already obtained;
• Pending approval requests;
• Intermediate reasoning results;
• Next-step routing.

2. External Side Effects (must have their own idempotency protocol)

• Payment debits;
• Order creation;
• Coupon issuance;
• SMS sending;
• Ticket creation;
• Third-party CRM writes;
• Calls to legacy systems with no queryable state.

These actions cannot rely on checkpoints alone for correctness.

Core Principle 2: Side Effects Must Have a Stable Logical Identity

The most practical approach isn’t asking “has this function executed before?” — it’s giving every business action a stable logical operation id.

import hashlib


def operation_id(run_id: str, step: str, business_key: str) -> str:
    raw = f"{run_id}:{step}:{business_key}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

Suppose the agent needs to create a refund request for order ORD-20260902-001. You can generate:

operation_id = operation_id(run_id, "create_refund", order_id)

Then use this same ID for:

• The downstream API’s Idempotency-Key;
• The unique key in the local side-effect ledger;
• The unique constraint in the Outbox table;
• The correlation field in logs and traces.

This way, even if the worker crashes and re-enters the same step after recovery, the system won’t mistake “the same logical action” for a brand-new business request.

Recommended Side-effect Ledger:

CREATE TABLE agent_side_effect (
    operation_id   VARCHAR(64)  PRIMARY KEY,
    run_id         VARCHAR(64)  NOT NULL,
    action_name    VARCHAR(128) NOT NULL,
    business_key   VARCHAR(128) NOT NULL,
    request_hash   VARCHAR(64)  NOT NULL,
    status         VARCHAR(32)  NOT NULL,
    external_ref   VARCHAR(128),
    result_json    TEXT,
    created_at     TIMESTAMP    NOT NULL,
    updated_at     TIMESTAMP    NOT NULL
);

Don’t write the execution logic like this:

result = external_api.create_order(payload)
checkpoint["order_id"] = result["id"]

The safer pattern is:

op_id = operation_id(run_id, "create_order", business_key)

cached = ledger.get(op_id)
if cached and cached.status == "SUCCEEDED":
    return cached.result

ledger.mark_started(op_id, request_hash(payload))
result = external_api.create_order(
    payload,
    idempotency_key=op_id,
)
ledger.mark_succeeded(op_id, result)
return result

The key point here isn’t the database table itself — it’s that the same operation_id can still be computed after recovery.

Core Principle 3: LLMs and External I/O Shouldn’t Enter Replayable Control Logic Directly

Temporal enforces this most explicitly: Workflow code must be deterministic, while LLM calls, tool calls, database access, and external APIs are all non-deterministic I/O and should go into Activities.

The reasoning isn’t complicated. Suppose recovery re-executes:

decision = llm.invoke(prompt)

The first time, the model outputs call_tool("refund"); the second time, due to sampling, model version, or server-side behavior changes, it might output call_tool("cancel_order"). At that point, the system isn’t “recovering” anymore — it’s generating a brand-new future from the middle of history.

The more sensible semantics for a durable runtime are:

Workflow / Orchestrator
   |
   +-- schedule LLM Activity
   |        |
   |        +-- historical result recorded
   |
   +-- replay uses recorded result

During recovery, reuse already-completed non-deterministic results instead of asking the model again. LangGraph’s Functional API follows a similar approach: results from Tasks and subgraphs can be persisted, and completed Tasks don’t need to be recomputed during recovery. The docs also explicitly require side-effect functions to remain idempotent.

Replay Contract: Code Upgrades Can Also Break Recovery

Durable Execution has another frequently overlooked problem: a running agent may span application version upgrades.

For example, a task starts on Monday:

planner -> lookup -> approval -> payment -> notify

On Tuesday, new code is released, changing it to:

planner -> lookup -> risk_check -> approval -> payment -> notify

On Wednesday, an old task resumes from the approval checkpoint. Are the new code topology, node identities, Task ordering, and historical checkpoint compatible? That’s the Replay Contract.

Microsoft Agent Framework’s checkpoint documentation explicitly notes that rehydration must preserve workflow topology and executor identity. LangGraph warns that when the ordering of Tasks or interrupts inside a node changes, recovery may fail to correctly match already-saved results.

So long-running agents shouldn’t just version their Prompts and Tool Schemas — they also need version governance for the workflow definition. At minimum, record:

run_id: agt_20260902_001
workflow_name: claims-agent
workflow_version: 7
checkpoint_schema_version: 3
tool_catalog_version: 18
prompt_release: prod-2026-09-02
model_policy_version: 11

On recovery, first perform a compatibility check — don’t just force the latest code to continue.

Engineering in Practice: Split an Agent Step into Four Phases

In production, you can split every step with side effects into four phases:

Prepare -> Execute -> Commit -> Checkpoint

• Prepare: compute the operation_id, request parameters, request hash, and current business preconditions. No irreversible side effects occur in this phase.
• Execute: actually call the external system. If the downstream supports idempotency keys, you must pass the operation_id; if it doesn’t, construct idempotent semantics as best you can using the local ledger + query API + unique business key.
• Commit: record the external reference, execution result, committed timestamp, and response hash. At this point, the local system knows the external action is complete.
• Checkpoint: finally advance the workflow state.

This ordering can’t eliminate all uncertainty windows in distributed systems, but it dramatically narrows the window where “the external system succeeded but the local system has no idea.”

The Trickiest Window: External Success, Crash Before Local Commit

This is the classic problem every durable agent must face:

Agent -> Payment API: charge()
Payment API -> success
Agent process crashes
Ledger.mark_succeeded() 尚未执行

After recovery, the local state still shows STARTED. The correct strategy here isn’t blind retry — it’s reconciliation:

1. If the downstream supports idempotency keys, retry with the same key;
2. If the downstream supports queries, look up by business key / external request ID;
3. If confirmation is impossible, mark the state as UNKNOWN;
4. For high-risk business operations, escalate to manual compensation rather than letting the agent guess.

Recommended state machine:

PENDING
   |
   v
STARTED
   ├──→ SUCCEEDED
   ├──→ FAILED_RETRYABLE
   ├──→ FAILED_FINAL
   └──→ UNKNOWN

The UNKNOWN state is critically important. For payments, claims, orders, coupon issuance, and similar operations, it’s far better to enter manual review than to incorrectly compress “I don’t know whether it succeeded” into “it failed.”

Retry Policy: Don’t Let Every Layer Retry on Its Own

Agent systems easily fall into retry storms:

HTTP Client retry 3 times × Tool SDK retry 3 times × Agent node retry 3 times × Workflow retry 3 times = worst case 81 calls

Production environments should designate a single retry owner:

Layer	Responsibility
HTTP SDK	Disable automatic retries
Tool Adapter	Error classification only
Durable Activity	Network-class retries
Workflow	Business-level compensation and state advancement

You also need to distinguish error types:

Retryable

• 429;
• Transient 5xx;
• Network timeouts;
• Temporary dependency unavailability.

Non-retryable

• Invalid parameters;
• Insufficient permissions;
• Content policy rejections;
• Explicit business rejections;
• Schema incompatibility.

This not only reduces failure amplification — it also makes historical records far easier to interpret.

Human-in-the-Loop: The Longer the Pause, the More Version Drift Matters

Human approval is one of the most typical use cases for Durable Execution. An agent might send an approval request today and receive the response next week. Microsoft Agent Framework’s checkpoints can save pending requests and re-emit the corresponding events on recovery.

But production systems should also save:

• The workflow version associated with the approval request;
• The business snapshot visible at approval time;
• A hash of the approved object;
• An expiration time;
• Whether business conditions need re-validation before recovery.

For example, a 500 RMB refund approved a week ago might no longer be valid if the order has since been manually processed. The agent shouldn’t blindly continue based on the old checkpoint. Therefore: checkpoints guarantee execution continuity; business precondition validation guarantees the semantics are still valid.

Production Monitoring: Don’t Just Monitor Agent Success Rate

Durable agents need at least the following categories of metrics:

Recovery Metrics

• workflow_resume_total
• checkpoint_restore_failure_total
• resume_latency
• checkpoint_age

Replay Metrics

• replayed_step_total
• task_result_reused_total
• workflow_version_mismatch_total

Side-effect Metrics

• side_effect_duplicate_prevented_total
• idempotency_key_hit_total
• side_effect_unknown_total
• reconciliation_total

Retry Metrics

• activity_retry_total
• retry_exhausted_total
• non_retryable_failure_total

The most important alert isn’t ordinary retries — it’s:

side_effect_unknown_total > 0

Because it means the system has entered the gray zone of “cannot confirm whether an external side effect completed.”

When to Use This

Durable Execution is especially well-suited for:

• Business agents spanning multiple external systems;
• Agents requiring human approval;
• Long-running tasks lasting minutes to days;
• Multi-agent collaboration workflows;
• Scenarios with side effects like payments, orders, tickets, and notifications;
• Environments where workers need elastic scaling and can restart at any time.

If the agent is just a single-request Q&A with no external writes and no need to pause and resume, introducing a full durable runtime may add unnecessary complexity.

Common Misconceptions

Misconception 1: Checkpoints mean exactly-once. No. Checkpoints solve state recovery; idempotency solves duplicate side effects. They are two different things.

Misconception 2: Recovery resumes from the last line of code. Most frameworks’ recovery semantics are closer to “re-execute from a stable boundary and reuse persisted results” — not process-level instruction pointer resumption.

Misconception 3: All APIs can be safely retried. For side-effect interfaces like payments, coupon issuance, and order creation, automatic retry itself is a risk source if there’s no idempotency semantics.

Misconception 4: Only agent state needs versioning. Long-running tasks also need version compatibility for Workflow Definitions, Tool Contracts, Prompt Releases, and Checkpoint Schemas.

Misconception 5: Agent Memory equals Durable Execution. Memory solves “what the agent remembers”; Durable Execution solves “how far the agent got, which actions have already happened, and how to continue after failure.” They serve different purposes.

Pre-Launch Checklist

Before production release, at minimum verify:

• Every irreversible side effect has a stable operation_id;
• Downstream systems support idempotency keys;
• Systems without idempotency support have a reconciliation path;
• Checkpoint recovery can’t re-enter a Node unintentionally;
• LLM/Tool/API calls are isolated into Tasks/Activities with persistable results;
• There’s a single retry owner;
• Retryable and non-retryable errors are distinguished;
• The UNKNOWN state has a manual handling process;
• Old checkpoints remain recoverable after workflow upgrades;
• Human-in-the-loop recovery re-validates business preconditions;
• Failure drills cover “external success, crash before local commit.”

FAQ

Q: Does checkpointing mean an agent gets exactly-once execution semantics?

No. Checkpoints save execution progress, but external side effects can occur between two checkpoints. To prevent duplicate actions, you still need idempotency keys, unique constraints, a side-effect ledger, or reconciliation.

Q: Why emphasize the Replay Contract?

Because a running agent may span code releases. At recovery time, if node identities, Task ordering, Workflow topology, or the Checkpoint Schema have changed, historical state may not be safely interpretable by the new version. The Replay Contract defines which changes are compatible and which require starting a new execution.

Q: Does Durable Execution make all agents more complex?

It adds infrastructure and state governance costs, so it shouldn’t be overused. Short requests, read-only tools, and side-effect-free scenarios can stay simple; long-running tasks, human pauses, and high-value side effects are where it’s most worth adopting.

References

1. LangGraph Functional API — Durable execution & idempotency: https://docs.langchain.com/oss/python/langgraph/functional-api
2. LangGraph Graph API — Re-execution and idempotency: https://docs.langchain.com/oss/python/langgraph/graph-api
3. LangGraph Interrupts — Side effects before interrupt must be idempotent: https://docs.langchain.com/oss/python/langgraph/interrupts
4. Temporal AI Agent Reference Architecture — Workflows orchestrate, Activities execute: https://go.temporal.io/platform-hub/ai-engineering/ai-reference-architecture
5. Temporal AI Engineering Patterns: https://go.temporal.io/platform-hub/ai-engineering/ai-patterns
6. Microsoft Agent Framework Workflows — Checkpoints: https://learn.microsoft.com/en-us/agent-framework/workflows/checkpoints
7. Microsoft Agent Framework — Durable Extension: https://learn.microsoft.com/en-us/agent-framework/integrations/durable-extension