Two weeks ago an agent I’d been running for forty-five seconds crashed because a downstream HTTP call timed out, the orchestrator silently retried from the start, and the second run hallucinated half of what the first run had already done — including writing a row to the production database that we then had to manually reconcile against the eval trace we’d lost. The fix was three lines of code in a checkpoint helper I should have written on day one. The lesson was that I had been thinking about agents like a prompt engineer for two years and like a distributed-systems engineer for ten, and the distributed-systems engineer had been quietly muttering at me from the back row the whole time.
This is the fifth post in an arc I’ve been writing. Context engineering was the runtime discipline — how you assemble the model’s inputs every turn. Spec-driven development was the design-time philosophy — your codebase isn’t your codebase, your specs are. Evals was the verification layer — if you ship AI features without evals you don’t have a product, you have a demo with users. OpenSpec was the operating system that runs all three on a brownfield team without collapsing. Every one of those posts assumed an agent that finishes the task. In production, agents don’t finish. They crash mid-tool-call. They exhaust the context window halfway through a long-horizon job. They get retried by an orchestrator that doesn’t know which side-effects already landed. They stall on a human-in-the-loop checkpoint that the human forgets about for three days.
Durable execution is the substrate underneath the other four disciplines. It’s the fifth one, and the arc was always incomplete without it.
Here’s the thing I’d ask any senior .NET engineer reading this to notice before the rest of the post even starts: you already understand durable execution. You shipped it. You shipped it in csharp-to-claude-agents-path territory before LLMs were a product. Every MassTransit saga you wrote was durable execution. Every MediatR pipeline with a compensation handler was durable execution. Every Hangfire job with an idempotency key was durable execution. Every NServiceBus message-broker contract that distinguished OnceOnly from AtLeastOnce was durable execution. The pattern is identical. The only thing that changed is that the participant is now non-deterministic and burns tokens. That isn’t a new problem — that’s a constraint on an old problem, and the old problem has solved patterns.
What “durable” actually means for an agent
A durable execution is one whose progress survives the death of the process running it. Crash the box, restart the orchestrator, swap the underlying model, time out the network — the workflow picks back up from the last committed checkpoint, does not double-spend, does not reorder side-effects, and arrives at the same outcome it would have arrived at had nothing failed.
Two things go wrong if you try to retrofit this onto an agent loop that wasn’t designed for it. First, agents are stochastic — replaying the same prompt at the same temperature against the same model often produces a different tool call, so naive “just retry the agent from the beginning” guarantees different side-effects on the second run. Second, agents are token-expensive — replaying a forty-thousand-token conversation history every time you resume is a budget question, not an engineering question, and the finance department will eventually notice. Durable execution for agents is the discipline of designing the loop so that neither of those things bites you in production.
The vocabulary lands cleanly on the workflow-engine concepts a .NET engineer already owns:
| Workflow / saga concept | Agent equivalent | What it costs you to skip |
|---|---|---|
| Idempotent activity | Idempotent tool call | Double-charged customers, duplicated database rows, the production reconcile pager I lit up two weeks ago |
| Determinism + replay | Checkpointed plan + replay from last committed step | Tokens you can’t expense; reasoning traces that look like a new agent each retry |
| Saga compensation handler | Tool-call rollback (refund, delete, retract) | Half-applied changes; manual ops cleanup; lost trust with downstream systems |
| Workflow-as-state-machine | Plan-as-state-machine, agent-as-participant | The orchestrator doesn’t know where the agent is in the plan, can’t recover, can’t observe |
OnceOnly semantics | tool_use_id-keyed dedupe at the activity boundary | Tools fire twice on retry; race conditions; eventual-consistency surprises |
| Human-in-the-loop checkpoint | Approval step modeled as a durable wait | Workflow gets stuck; the human forgets; the run dies; you lose three days of state |
If you read that table and felt nothing new — that’s the point. The Anthropic Agent SDK ships a durable run primitive that exposes exactly this shape, with tool_use_id dedupe, plan-step checkpoints, and resumable conversation state across vendor-side retries. The Inngest Agent Kit, Temporal’s AI extensions, Restate’s agent runtime, Cloudflare Workflows, and AWS Step Functions all converged on roughly the same primitive over the last twelve months. The reason the convergence happened so fast is that the underlying pattern was already standard outside AI. The vendors are catching up to what the saga-orchestration crowd has been doing since the early 2010s.
The four failure modes of non-durable agents
Every agent that hasn’t been designed for durable execution dies one of four ways in production. I have a scar from each.
1. Mid-tool-call crash. The agent calls a tool that takes ten seconds to return. At second six, the container is preempted, the function times out, or the model API itself returns a 502 from an upstream blip. Did the tool’s side-effect land or not? If you don’t have a tool_use_id-keyed dedupe table on the receiving side, you have no way to know. You can rerun and double-charge. You can not-rerun and silently lose work. There is no correct answer at this layer — the correct answer was making the tool idempotent at design time. If you only learn one rule from this post: every tool that mutates state outside the agent’s own context must accept an idempotency key, and the receiving system must check it before applying the change.
2. Context-window exhaustion mid-run. The agent is forty-five turns into a long-horizon task. The system prompt is two thousand tokens; the accumulated tool-call history is forty thousand. The next planning step needs eight thousand tokens of reasoning. The model returns a refusal, a truncated plan, or — worse — a confident plan based on a context the model has silently summarized at the wrong layer. This is what Context Engineering calls “running out of working set,” and it’s where most production agents stall. The fix lives at the runtime layer: aggressive history compaction at well-defined checkpoint boundaries, with the compacted summary written to durable storage so a resumed run rehydrates the same compacted state instead of replaying the raw history.
3. Retry storm on non-idempotent tools. The agent fails midway. The orchestrator retries. The retry hits a tool the previous run already called. The tool isn’t idempotent. The second run spawns a third side-effect. By the time you notice, you have three Stripe charges, two Slack messages, and one production database row that exists in triplicate. The retry mechanism made it worse, not better — the agent never recovered, the orchestrator just multiplied the damage. This is the failure mode that taught the saga-orchestration community to bake idempotency keys into the contract, not bolt them on after.
4. Stalled human-in-the-loop pauses. The agent reaches an approval checkpoint. It pings Slack. The human is at lunch, or on a flight, or — most often — opens the message, decides to respond later, and forgets. The agent’s process keeps running, waiting on a synchronous response, burning resources, holding open a network socket. Or worse, the agent’s process exits while waiting, the human approves three hours later, and the response goes into a Slack thread no listener is reading. The pattern in non-LLM saga territory is to model the human-in-the-loop pause as a durable wait — the workflow goes to sleep, the resume signal is a durable callback, and the workflow doesn’t care whether the resume happens in three seconds or three days. The same pattern transfers directly.
Each of these is a runtime-substrate failure, not a model failure. Throwing a smarter model at any of them doesn’t fix any of them.
Five rules I’ve earned shipping durable agents
These are the rules I’d write into the project.md of any new repo that ships agents in 2026. They are not original — they’re what the workflow-engineering community has been saying for a decade, restated in agent vocabulary.
Rule 1: Idempotent tools by default
Every tool that mutates state outside the agent’s local context accepts an idempotency key as a first-class parameter, and the receiving system checks it before applying the change. The key is the tool_use_id from the Anthropic/OpenAI tool-call schema in the simplest implementations; for tools that span multiple turns or compose other tools, it’s a stable hash of the plan-step ID. There is no exception to this rule. The cost of writing it is one extra parameter on every tool; the cost of skipping it is the production reconcile I started this post with.
Why this is non-negotiable. Tools that look “read-only” today get retrofitted with caching or telemetry tomorrow, at which point they become side-effecting in a way the agent loop can’t see. Make the entire tool surface idempotent at the contract level and you never have to relitigate which tools “are or aren’t” — they all are.
Rule 2: Checkpoint after every external write
The agent’s plan is divided into steps. After every step that produces an external side-effect — a database write, an API call to a third party, a message to a queue — the orchestrator commits a durable checkpoint. The checkpoint contains: the step ID, the tool call that ran, the result returned, and a hash of the agent’s working context at that moment. On resume, the orchestrator replays the plan from the last committed checkpoint, not from the start.
The checkpoint store doesn’t need to be sophisticated. A Postgres table with a unique constraint on (workflow_id, step_id) is enough. The Cloudflare Workflows API exposes this as step.do(). The Anthropic Agent SDK exposes it as the checkpoint callback. Temporal exposes it as activity history. Pick one and standardize on it across the codebase.
Rule 3: Never resume into a stale plan
The hardest bug in durable agents is the resume-into-staleness bug. The agent crashed at step seven of a twelve-step plan. The plan was generated from a context that included data — say, an inventory snapshot — that was fresh at the time. Three hours later the workflow resumes. The inventory snapshot is now wrong, but the plan doesn’t know that, and the agent proceeds confidently from step eight against a world that’s changed.
The rule is: any plan element that depends on external state has a freshness budget, and resume past the budget triggers a re-plan from the last good checkpoint, not a blind continuation. The freshness budget is part of the plan, not part of the runtime config. The agent decides what’s fresh; the orchestrator enforces it.
Rule 4: Budget tokens on resume, not just on start
A resumed agent inherits a context window the original run grew. If you don’t compact aggressively at checkpoint boundaries, every resume is more expensive than the previous one, and a workflow that retries four times might cost more in tokens on the fifth try than it cost across the first four combined. This is the failure mode that turns “well-tuned eval suite” into “finance ticket.”
The discipline is to compact the conversation history at every durable checkpoint — fold the raw turns into a structured summary that lives in the checkpoint payload, and rehydrate from the summary on resume rather than from the raw turns. The Claude API makes this easy with prompt caching: the checkpoint summary becomes the cacheable prefix, and only the new working turns sit outside the cache. The eighty-percent cost lever I called out in csharp-to-claude-agents-path is exactly this lever — most teams just don’t apply it to resumed runs, only to fresh ones.
Rule 5: Treat the workflow as the source of truth, the agent as its participant
This is the rule that the saga-orchestration community internalized fifteen years ago and the LLM ecosystem is still catching up to. The workflow is not “the thing the agent runs”; the workflow is the contract, and the agent is a participant in it. The agent can be replaced with a different model, a different vendor, or a deterministic function, and the workflow keeps working. The state lives in the workflow; the agent has no durable state of its own.
If you find yourself reasoning about “what the agent knows at step seven” or “what the agent remembers from step three,” you have already lost. The agent knows what the workflow tells it at step seven, period. The agent remembers what the checkpoint payload contains at step three, period. The agent’s “memory” is the workflow’s state, materialized into the prompt at each step. That’s the entire mental model.
Where this still falls apart
The honest part. Durable execution solves three of the four production-agent failure modes cleanly. The fourth — and a few adjacent ones — are still open problems that I do not have clean answers for. If you read the next four paragraphs and have answers, write them up and email me; I will link to your post.
Long-horizon human-in-the-loop is a UX problem disguised as an infrastructure problem. Durable waits handle the workflow-side correctness — the workflow sleeps, the resume signal lands, the workflow continues. But the human side is unsolved. Three days into a paused workflow the human has forgotten the context that made the approval question meaningful. Re-presenting the context on resume is a generation problem, not a retrieval problem; you have to re-explain the workflow to the human, not just show them where it stopped. I have not seen anyone solve this well in 2026; the closest is treating the resume notification as a separate generation pass that synthesizes the why-this-matters context fresh.
Cross-vendor model swap mid-run is theoretically possible, practically painful. You can compact the context, persist it, and rehydrate it against a different model on resume — Opus to Sonnet for cost reasons, or Claude to GPT for capability reasons. In practice the prompt-format differences, the tool-call schema dialect differences, and the model-specific behavioral quirks mean the resumed run behaves differently even though the context is “the same.” The infrastructure can swap models. The agent might not survive the swap. I currently treat mid-run vendor swaps as a code smell, not a feature.
LLM-as-judge retry semantics are not idempotent. Evals at the LLM-judge layer are non-deterministic by construction. If your durable workflow retries the judge step, the second judgment can disagree with the first. The honest answer is to either snapshot the judgment in the checkpoint payload (so retry replays the cached judgment, not the model) or to commit at the level of “we asked the judge once,” with all the failure-mode implications that brings. I currently do the former and I’m not happy about it.
Cost of the durable-execution infra itself. Every checkpoint is a write. Every resume is a read. Every retried workflow pays the storage and read cost twice. For most production agents the overhead is in the single-digit-percent range, but for high-fanout multi-agent systems the math gets harder. I haven’t seen a clean treatment of “when is durable execution too expensive to be worth it”; my heuristic is that any workflow with an external side-effect is worth checkpointing and any workflow that only generates text without side-effects probably isn’t.
The career angle: durable-agent engineer is the next named role
The pattern across the pillar arc has been the same. Context engineering became a named role; eval engineer is becoming one. Spec author is the next role the brand is positioning for. Durable-agent engineer is the next role after that, and it’s the one with the deepest moat for a senior .NET background.
Junior (zero to two years out of school). Learn the tools. The Claude Agent SDK and the Anthropic SDK ship the durable-run primitives at the surface; reading their docs is the cheapest way in. If you don’t have a .NET or distributed-systems background, do the RAG and Agents course — it covers the agent-loop fundamentals you need before durable execution makes sense as a concept. Build one small durable agent end-to-end (workflow + idempotent tool + checkpoint store + resume) and you’ll have a portfolio piece that almost no junior engineer in 2026 actually has.
Mid (three to seven years). Own the substrate. Pick the orchestrator your team will standardize on — Temporal, Inngest, Restate, Cloudflare Workflows, or the Agent SDK’s native runtime — and own it across the codebase. Write the conventions: which idempotency-key shape, which checkpoint payload schema, which retry policy. The mid-level engineer who quietly authors the team’s AGENTS.md durable-execution section is the one who becomes irreplaceable in eighteen months. If your background is .NET, the AI-Powered .NET Development course is where the agent-loop / saga-orchestration / ASP.NET-Core integration story lands as a single thread; it’s the natural next step after the standard backend curriculum.
Senior and staff (eight years and up). Treat durable execution as an architectural concern, not a library choice. The interesting questions at this rung are: how do durable agents compose with the existing message-broker contracts? How do you migrate a brownfield agent codebase from “the agent loop runs in-process” to “the agent loop is a workflow”? How do you set a token budget that survives retries? How does the durable-execution layer interact with the evals layer when a judgment is itself a workflow step? These are the questions every senior .NET engineer migrating to AI engineering has prior art for; the recognition that the prior art transfers is the high-leverage move. The Spec-Driven Development and OpenSpec Mastery courses give you the methodology framework to make those decisions defensible to the rest of the team.
Closing the arc
The arc was always five disciplines, not four. Context engineering is how you assemble the inputs. Spec-driven development is how you express the intent. Evals are how you prove the output is what you wanted. OpenSpec is the operating system that runs all three on a real team. Durable execution is the substrate that makes all of it survive Tuesday afternoon when the network blips and the model returns a 502 mid-tool-call and the human goes home before approving the checkpoint.
Every senior .NET engineer reading this post has already shipped the pattern. The only thing left to do is name it, and then build it into the agent loop on day one instead of bolting it on after the first production incident. The cost of writing the rules into a new agent codebase is a day. The cost of retrofitting them after the first reconcile pager is a week of cleanup and a slow recovery of trust with the systems you write into.
The substrate is durable. The agent is its participant. The workflow is the truth. Specs, context, evals, OpenSpec, and durable execution — that’s the five-discipline arc. Everything else is interface over it.
