Part 1 of 2: Architecture: state model, execution model, reliability primitives
The Agent Is the Easy Part. Agent State Management Is the Hard Part.
You built the agent over a weekend. It works great in demos. Then you put it in front of a real user on a real task and something with 9 steps, 4 tool calls, and a flaky third-party API.
It gets to step 6. A timeout fires. The tool call hangs. The orchestrator retries.
And then restarts. From step 1.
The user gets a half-finished task, a duplicate calendar invite, and no explanation. You get a bug report that says, “It just stopped working.”
You debug for two hours and find nothing wrong with the prompt. Nothing wrong with the model. Nothing wrong with the tool logic.
The problem was simpler and more fundamental: you had no state. When the crash happened, the agent had nowhere to land. So it fell all the way back to the beginning.
This is the failure mode nobody talks about when they demo agents. But it’s the one that actually kills them in production.
What “State” Really Means
When people say “agent state,” or “agent state management”, they usually mean conversation history. That’s one piece but it’s the smallest piece.
Real agent state is everything the system needs to know to answer one question: where are we, and what happened to get here?
That breaks down into five layers:
Task position: which step the agent is on, which steps are complete, and which are pending. Not “the agent is running,” but “the agent is on step 6 of 9; steps 1–5 are committed.”
Tool outputs: what each tool call returned, verbatim. Not a summary. The raw output is attached to the step that produced it.
Decisions and rationale: what the agent chose at each branch point and why. This sounds like a nice-to-have. It isn’t. Without it, you can’t audit failures, you can’t reproduce edge cases, and you can’t explain to a user why the agent did what it did.
Retry context: how many times each step has been attempted, what the backoff state is, and what errors were encountered. Without this, your retry logic has no memory. It will retry indefinitely, or not at all, with no awareness of what it already tried.
Failure context: when something breaks, the full error: timestamp, step, tool, input, output, and exception. Not a logline. Structured data attached to the state record.
If any of these layers is missing, your agent is flying partially blind. When it crashes, you’re debugging with incomplete information or none at all.
Stateless vs. Stateful: The Real Tradeoffs
| Stateless | Stateful | |
|---|---|---|
| Time to build | Fast | Slower |
| Demo quality | Great | Same |
| Failure recovery | Restart from zero | Resume from last checkpoint |
| Debugging | Hard, reconstruct from logs | Structured, inspect state snapshot |
| Duplicate tool calls | Likely on retry | Preventable with dedup |
| Cost on failure | Full re-run | Partial re-run |
| Operational complexity | Low | Higher |
The trap is that stateless agents look fine until they don’t. A 3-step task with no retries in a controlled demo will never expose the problem. A 12-step task with one flaky API call in production will expose it immediately.
When stateless is acceptable: short tasks (2–3 steps), low-stakes outputs, tasks where restarting from scratch is cheap and invisible to the user, and prototypes.
When stateful is required: any multi-step task where partial completion has side effects (emails sent, records created, payments initiated), any task that calls external APIs with rate limits or costs, and anything user-facing where “try again from the beginning” is unacceptable UX.
Most production agents need to be stateful. Most aren’t.
Durable Execution Patterns
The cleanest mental model for stateful agents: treat the agent as an event log, not a process.
A process disappears when it crashes. An event log doesn’t. Every action the agent takes, every decision, every tool call, every retry is appended as an immutable record. The agent’s “current state” is derived by replaying the log.
This gives you three properties that are very hard to get any other way:
Replay: if the process crashes, restart it and replay the log to restore exactly where you were. No manual reconstruction.
Compensation: if a step fails after side effects have already occurred (a tool call that partially succeeded), the log tells you exactly what happened so you can undo or compensate cleanly.
Auditability: the full history of every decision the agent made is queryable. Not inferred from logs. Structured, attached to the run.
In practice, this pattern is implemented in two ways:
Event log directly: you own the schema, you write the append logic, and you build the replay. More control, more work. Good when you need tight integration with your own infrastructure.
Workflow engine: tools like Temporal, Inngest, or similar give you durable execution as a managed primitive. Your “agent” becomes a workflow function. Retries, timeouts, and state persistence are handled by the platform. Less control, much less work. Usually the right starting point.
The key insight either way: the agent’s logic and the agent’s state must be separated. The logic is stateless code. The state is durable data. If you mix them, if the state lives in memory inside the running process, every crash destroys it.
Tool Calls Are Distributed Systems
Here is a thing that is easy to forget: every time your agent calls a tool, it is making a distributed system call.
Distributed system calls fail. They time out. Partially succeed. They succeed, but the response gets lost. They get retried and then succeed twice.
This means every tool call needs an idempotency key, a stable identifier that lets the tool (or your infrastructure) detect duplicate calls and return the original result instead of executing again.
The key needs to encode enough context to be unique per action but stable across retries of the same action:
{task_id}:{step_number}:{tool_name}:{args_hash}For example:
task_abc123:step_6:send_email:f3a9b12cIf the agent retries step 6, it uses the same key. If the email was already sent, the tool (or your deduplication layer) returns the original result. The user does not get two emails.
Without idempotency keys, you are assuming that every tool call will succeed exactly once, with no retries, and that timeouts will never occur. That assumption is false in production.
Your agent should always assume at-least-once delivery. Tool calls may execute more than once. Design accordingly.
Debugging: Logs vs. Snapshots
Logs tell you what happened. Snapshots tell you what the agent knew at the moment something happened and that is a different question.
A log line that says tool_call_failed: send_email step_6 tells you the tool failed. It does not tell you what inputs the agent passed, what the agent’s task context was, what it had already decided, or what it was planning to do next.
A state snapshot before and after step 6 tells you all of that.
The snapshot pattern:
Before each step begins, serialize the full agent state to your state store. After each step completes (success or failure), serialize it again. Tag each snapshot with the step number, timestamp, and outcome.
When something goes wrong at step 6 of 12, you can:
- Load the pre-step-6 snapshot and see exactly what the agent knew.
- Compare it to the post-step-6 snapshot to see exactly what changed (or didn’t).
- Replay from any prior snapshot to reproduce the failure in isolation
This is “time travel debugging”, the ability to restore the agent to any prior state and examine it. It is the single most powerful debugging capability for multi-step agents, and it requires almost no additional logic if your state model is already clean.
The cost is storage. The payoff is hours of debugging time saved on the first production incident.
The Anti-Hype Principle
The agent demo is easy to make impressive. A well-prompted model calling a few tools, happy path only, no retries, no failures, no concurrent users. It looks like magic.
Production is not the demo. Production has timeouts, flaky APIs, partial failures, users who interrupt tasks midway, and infrastructure that restarts at 3am.
The difference between an agent that survives production and one that doesn’t is almost never the prompt or the model. It’s whether someone built the state layer for better agent state management.