Why Long-Running Agents Need Durable Execution

Picture an agent that has been triaging a backlog for 55 minutes. It has read hundreds of documents, filed a dozen tickets and sent a few emails. Then a node upgrade evicts its pod. You restart it, and it begins at minute zero: same documents, a new plan, and the same emails going out again.

Durable execution for agents fixes that: each step's result is saved before the next one starts, so a crash costs you one step instead of the whole hour.

Why crashes hurt agents more than ordinary jobs

Crashes are routine: deploys, evictions, out-of-memory kills, reclaimed spot capacity. A batch job just reruns. An agent has three extra problems.

• Every step costs money. A rerun pays for the same model calls and tool calls twice.
• The model is not deterministic. The second attempt makes a different plan, so the first attempt's half-finished work no longer fits.
• Tools have side effects. The first attempt's emails, tickets and database writes are still out there.

What durable execution for agents actually means

Three rules. Record each step's result before moving on. On restart, replay the record instead of redoing the work. Make the one step that was in flight when the crash hit safe to run twice.

Workflow engines and checkpointing libraries implement these rules. LangGraph's durable execution docs say you need a checkpointer and a thread ID per run, and you resume by invoking the graph again with the same thread ID. Temporal records the result of each activity (its term for a retryable unit of work, such as a model call or tool call) and rebuilds the workflow from that history instead of re-running completed work.

A checkpoint-and-resume sketch

Every model call and tool call goes through one wrapper that checks a table first.

import json
import sqlite3

db = sqlite3.connect('agent_runs.db')
db.execute(
    'CREATE TABLE IF NOT EXISTS steps ('
    'run_id TEXT, seq INTEGER, result TEXT, '
    'PRIMARY KEY (run_id, seq))'
)


def durable(run_id, seq, fn):
    row = db.execute(
        'SELECT result FROM steps WHERE run_id = ? AND seq = ?', (run_id, seq)
    ).fetchone()
    if row:
        return json.loads(row[0])  # finished before the crash: replay it
    result = fn()
    db.execute('INSERT INTO steps VALUES (?, ?, ?)', (run_id, seq, json.dumps(result)))
    db.commit()
    return result


def run_agent(run_id, goal, call_model, call_tool):
    messages = [{'role': 'user', 'content': goal}]
    seq = 0
    while True:
        reply = durable(run_id, seq, lambda: call_model(messages))
        seq += 1
        messages.append({'role': 'assistant', 'content': reply['text']})
        if not reply.get('tool_call'):
            return reply['text']
        call = reply['tool_call']
        key = f'{run_id}:{seq}'  # stable across restarts
        result = durable(run_id, seq, lambda: call_tool(call['name'], call['args'], key))
        seq += 1
        messages.append({'role': 'user', 'content': json.dumps(result)})

Run it with a fixed run ID, kill it mid-run, then start it again with the same ID. Finished steps come back from SQLite, and because the model's replies are replayed rather than regenerated, the message history rebuilds identically.

The tool call in flight at the crash may already have run, so it gets a key built from the run ID and step number, stable across restarts, that the tool can use to deduplicate.

Use a workflow engine for agents once the sketch runs out

The sketch has no scheduler, worker pool, timeouts or view into stuck runs. A workflow engine supplies those. Below is the same loop as a Temporal workflow in Python; per Temporal's activity timeout docs, you set a Start-To-Close timeout on each activity and pass a RetryPolicy to execute_activity.

from datetime import timedelta

from temporalio import activity, workflow
from temporalio.common import RetryPolicy


@activity.defn
async def call_model(messages: list[dict]) -> dict:
    ...  # your LLM call


@activity.defn
async def call_tool(call: dict) -> dict:
    ...  # your tool call, written to be safe to run twice


@workflow.defn
class AgentRun:
    @workflow.run
    async def run(self, goal: str) -> str:
        messages = [{'role': 'user', 'content': goal}]
        while True:
            reply = await workflow.execute_activity(
                call_model,
                messages,
                start_to_close_timeout=timedelta(minutes=2),
                retry_policy=RetryPolicy(maximum_attempts=5),
            )
            messages.append({'role': 'assistant', 'content': reply['text']})
            if not reply.get('tool_call'):
                return reply['text']
            result = await workflow.execute_activity(
                call_tool,
                reply['tool_call'],
                start_to_close_timeout=timedelta(minutes=5),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
            messages.append({'role': 'user', 'content': str(result)})

The workflow function is re-executed from its history, so keep it deterministic: model calls, tool calls, clock reads and random numbers all belong in activities.

The LangGraph equivalent: compile the graph with a checkpointer, then resume with the same thread ID.

config = {'configurable': {'thread_id': 'invoice-run-4812'}}
graph.invoke(None, config)  # None input: continue from the last checkpoint

Is a retry loop enough?

For a call that fails while your process is alive, yes: wrap it in exponential backoff. But when the process itself dies, the retry loop and its memory of what it already did die with it, and rerunning from the top brings back repeated spend, a different plan and repeated side effects. Retries handle a flaky call. Durable execution handles a flaky host.

The fair counterpoint: if your agent runs for twenty seconds and only reads data, a retry loop is fine. Durability earns its keep once runs last minutes or hours, or touch anything you cannot undo.

Know the limits before you adopt it

• At-least-once, not exactly-once. A crash between a tool finishing and its result being saved re-runs that tool. Idempotency covers the gap.
• Resume is not line-exact. LangGraph's docs say resumption picks a suitable restart point rather than the exact line where execution stopped, so wrap side effects in tasks.
• Histories grow. Temporal limits the size of a workflow's event history, so very long loops need Continue-As-New.

First step to take

Pick your longest-running agent. Log every model and tool result to a table keyed by run ID and step number, then kill -9 it halfway through a run and watch the restart. If it starts over, you have your case for a workflow engine.

In production, a crash at minute 55 should be a minor incident, not a customer-facing one.