Why Taipei Cloud Servers Are the Ultimate Sol
As digital transformation accelerates, businesses in th...




A checkpoint records where an agent was, not what it did. Restore one after a crash and you get the position back, not the world. The payment that cleared, the ticket that was filed, the message that was sent, the row that was committed — every one of those happened in the seconds between the last checkpoint and the failure. The agent wakes up with a stale copy of its own memory and no idea any of it took place, so it does the reasonable thing: it does them again.
This is not a bug in any particular framework. It is the arithmetic of two independent commit paths that were never designed to agree. Your agent commits its state on one schedule. The services it writes to commit their effects on another. Recovery asks one of them to stand in for the other, and it cannot.
The failure surfaces as a duplicate. A second charge. A second deployment. The same webhook delivered to a customer twice. The agent reports success, because from its point of view it succeeded. The accounting is off by exactly the work that fell inside the window.
The reason this is hard to see is that checkpoint-restore works beautifully in a development environment. You kill the process, you restart it, the graph resumes from the last super-step, and the run completes. The demo passes. Nothing in it touches an external system, so there is no second world to disagree with.
Every durable-execution system you can read about starts from the same premise. Temporal's documentation describes durable execution as the ability of a workflow to maintain state and progress through failures, achieved by an event history that records the state of the execution at each step, so that the run can resume from the last recorded event. LangGraph's persistence layer describes checkpointers as persisting a thread's graph state as checkpoints, with a separate store for application data that lives outside the graph state. Both are accurate descriptions of what the mechanism does. Neither claims to describe what happened in the world.
Read the words carefully. The event history records the state of the execution. The checkpointer persists graph state. State is the agent's belief. It is a record of what the agent knew, not a record of what it caused. A resume-from-last-event is a resume of a belief, and beliefs can be wrong in exactly one direction that matters: they can be behind reality.
Put a timeline on it. A tool call leaves the agent, arrives at a service, and the service commits an effect. The agent then records that the call returned. Two writes, to two different systems, at two different moments, and the failure can land between them.
Only the second case has a clean answer. The first and third are the same failure seen from two ends: an effect exists in the world whose existence the agent's state does not reflect, and the agent's state cannot be repaired by replaying its own history, because that history never contained the fact in the first place.
There is a related trap that follows from the same gap. A checkpoint can restore the agent's state while it is still behind the world, and then the agent does not replay the same request — it generates a new one. A model asked to redo a step does not reproduce its previous tokens; it produces a semantically similar request with different wording, a different idempotency token, a different amount. A service that keys deduplication on the exact request body sees a new request and executes it. The March 2026 preprint on this, which introduced the term semantic rollback attacks, is an early warning rather than settled evidence, but its mechanism needs no statistical support to be true: if the retry is regenerated rather than replayed, the request that arrives is not the request that landed.
Before any retry decision, an agent has to establish one of three outcome states for the action it is about to repeat. The checkpoint tells it none of them.
| Question | What the checkpoint says | What the service knows |
|---|---|---|
| Did the write commit? | Nothing. The call was in flight, so no result was recorded. | Yes or no, but only if you ask it in the right way. |
| How many times did it commit? | Nothing. Nothing was acknowledged, or one acknowledgement arrived. | One, twice, or partially, depending on the transport. |
| Was it the same request? | Nothing, once the agent regenerates it. | Only if the request carried a stable key. |
Notice where the answers live. Two of the three are only knowable by the service, and the third is only knowable if the request was designed to be recognisable. This is why the fix is not a better checkpoint. It is a contract between the agent and the services it writes to, and the checkpoint sits outside it.
This is no longer a hypothetical. A September 2026 study built a deterministic sandbox of six services with realistic contracts — optional idempotency keys, eventually consistent reads, and deliberately missing read paths — injected twelve fault modes at the service boundary, including late commits, redelivery, and partial batches, and graded every episode against a ledger of effects that had actually committed. Across 25,930 episodes, nine recent models, three production agent harnesses, two contract variants, and fifteen recovery conditions, the results separate cleanly by whether the agent could find out what happened.
| Condition | Duplicate rate | What predicts the outcome |
|---|---|---|
| A read-back immediately reveals the result | 0.5% for frontier models instructed to act exactly once | The model. It explains 53% of the explained variance; weaker models duplicate far more often. |
| The request is still in flight | 56% for the same frontier models | The contract. It explains 81% of the explained variance. |
| The transport delivered the request twice | 74% for the same frontier models | The contract, again. The agent never saw the second copy. |
Three findings in that set are worth more than the headline rates.
The first is that no verification-only policy can be exactly-once under late commits without a bound on how long a request may remain in flight. Waiting seems like the safe answer, and it works when the in-flight delay is short and known. But with heavy-tailed delays, even an hour of waiting per episode falls short of simply attaching an idempotency key to every write.
The second is the size of that difference. Offering an idempotency key on every write lowered the duplicate rate from 28% to 4%, in the paper's own comparison, because agents actually use keys when they exist. That is a seven-fold reduction from a contract change, not a model change.
The third is the one that should change how you monitor. The harness barely mattered — a guard that attaches keys transferred across harnesses unchanged. And the agents reported success in 90% of the episodes in which they had duplicated an effect. Your success metric is not merely blind to this failure; it is systematically wrong about it, because the agent's own report is generated from the same stale state that caused the duplicate.
The instinct when a checkpoint restores is to let the run continue from where it stopped. That is the wrong default, because it silently chooses replay over fork for every action in flight, and replay is only correct for the actions that never committed.
An October 2026 study formalises the test. It injects a controlled error, then replays the same downstream actions from the recovered state and from the target state, and requires that identical downstream actions produce identical outcomes — the paper calls this transition-consistent recovery. If the two runs diverge, the agent did not recover, whatever the final score says. Its finding is that current computer-use agents fail to exploit the recovery capability they already have, and that accurate model-based recovery produces measurable end-to-end gains. The practical version for anyone running agents in production is smaller and sharper: recovery is a state you can check, not a success rate you can infer.
So make the branch explicit. After a restore, the recovery path should decide, per pending action, whether to replay it, skip it, or escalate — and that decision should be recorded. The mechanism that enforces it is a replay-or-fork choice at the point of restore, not an automatic replay of every tool call in the pending list.
A deterministic program that replays an operation produces the same operation. The request bytes are identical, the key is identical, and a receiving service that has seen it before can recognise it.
An agent does not replay. It re-plans. Given the same restored state and the same goal, a language model regenerates the step, and the regenerated step is not byte-identical to the original. Sometimes that is harmless, because the action is idempotent by nature — a read, a lookup, a status check. Sometimes it is the entire problem, because the action is a write, and the service has no way to know that this new-looking request is a repeat of one it already committed.
This is why idempotency keys matter more for agents than for ordinary services, and why the keys have to be stable across the regeneration. A key derived from the request body fails the moment the model rewords the request. A key derived from the intent — the action type plus the logical target plus the task — survives the reword, because it was never a function of the wording.
None of this would matter if crashes were rare. They are not, and the reason is that modern agent hosting is built around ceilings the agent does not control. These are not edge cases; they are defaults.
| Layer | Default | Consequence for an in-flight action |
|---|---|---|
| Managed agent runtime | Idle session timeout 900 seconds (15 minutes); maximum session lifetime 28800 seconds (8 hours) for microVM runtimes, up to 1209600 seconds (14 days) for capacity-provider runtimes | The instance is terminated at the ceiling. Termination itself can take up to 15 seconds for logging and process completion. The session can continue on a newly provisioned instance — which is exactly the restore. |
| Container orchestrator | Graceful termination default 30 seconds; then SIGTERM to PID 1, then SIGKILL to whatever remains | One tool call that takes 60 seconds overruns the window twice over. If the kubelet or the runtime restarts while waiting, the cluster retries from the start, including the full original grace period. |
| Service manager | Restart defaults to no; restart delay defaults to 100 ms; start rate limiting defaults to 5 starts within 10 seconds |
A framework with no restart policy simply stops. With one, a crash loop is capped: more than 5 starts in 10 seconds and the manager stops trying, and only a later manual or timer start, or a reset of the failed state, reactivates the restart logic. |
| Process signal | SIGKILL cannot be caught, blocked, or handled | No shutdown hook runs. Anything that was going to be recorded before exit is not recorded. |
Two of these deserve emphasis. The start rate limit is the one people miss, because it turns recovery into a finite resource: a service that crashes and restarts every two seconds will exhaust its five-start allowance inside ten seconds and then stay down. And the 8-hour ceiling means that for long tasks the restore is not a rare event at all — it is part of the normal operating cycle.
Here is the arithmetic that makes the whole problem routine rather than theoretical. Current frontier models complete a fifty-percent-reliability task of roughly twelve hours, according to the 2026 METR time-horizon measurements as reproduced in the September 2026 State of Open Source AI report, which also notes that under eight hours models of both open and closed classes can generally do the job, and past twelve neither can. The default maximum session lifetime for a microVM-class managed agent runtime is eight hours.
Twelve divided by eight is one and a half. A task at the edge of what the model can do does not fit inside a single session, by design, and the boundary is enforced whether or not the agent is anywhere near finished. Every crossing is a restore, and every restore is a replay decision for whatever was in flight.
Run the same numbers at scale. A single long-lived agent on a managed runtime that hits its ceiling every eight hours is restored 90 times a month and 1,095 times a year. At each of those restores, any action that was mid-flight is a candidate for either a lost effect or a duplicate one. This is not a tail risk. It is the median path of any agent that runs continuously, which is precisely what a 24/7 agent is. The related failure modes compound here rather than in isolation: a service that dies in week three of continuous operation, or a memory footprint that grows across restores, will also be restarting on this schedule.
The correction is a change of architecture, not of library. Keep the checkpoint for what it does well — restoring the agent's position, its plan, its conversation, its accumulated context. And put the record of effects somewhere the agent's own state cannot rewrite it.
You can test the window in the time it takes to make coffee, and it is the one test most retry suites skip. Ordinary retry tests inject the failure before the tool runs. The interesting window is after the tool runs and before the state commit, and it needs to be provoked deliberately.
If the first count is one and the second two, you have found the gap. It will not appear in any test that only fails before the tool runs, and it will not appear in any monitoring built on the agent's success rate.
The recovery discipline above assumes something about the host: that the process is the agent's own, that you can send it a signal, watch it die, restart it on the same disk, and inspect what survived. You cannot run a ten-minute crash drill against a laptop that suspends, or against an environment where the process is managed by someone else's scheduler with rules you cannot see. The drill is only meaningful on a host you control, where a killed process leaves its checkpoint and its ledger exactly where the next start will look for them.
That is a modest requirement, and it is worth satisfying before the architecture rather than after it. A short-lived trial instance is enough room for the drill: a real environment with the same image, the same disk behaviour, and the same ability to be killed and restarted that production has. SurferCloud's UHost trial plan gives a 3-day or 7-day instance — 2 vCPU and 2 GB from $1.9 for 3 days or $3.9 for 7 days, on the same console and API as a full cloud server — which is exactly the kind of disposable host where you can kill the process in the dangerous window on purpose, as many times as it takes to see the duplicate. Full pricing is on the UHost trial plan page, and the same drill applies unchanged to a longer-lived instance.
The wider point is that crash recovery is not a property your framework can give you, because it is not a property of one system. It is a property of the agreement between the agent and everything it writes to. The checkpoint holds one half of that agreement, and the half it holds is the half that is allowed to be wrong.
It makes the agent's state recoverable, which is a different claim. The documentation for these layers is careful about this: a checkpointer persists a thread's graph state, and a separate store holds application data outside the graph state. Neither is a record of what the agent caused in other systems. Crash-safety for side effects comes from the contract with those systems, not from the checkpointer.
It is safe when the maximum in-flight delay is short and known. The measurement above shows that with heavy-tailed delays, waiting cannot substitute for keys — even an hour of waiting per episode fell short in that comparison. If you cannot bound the delay, bound it or key the write.
That is the strongest fix where you control the service, and it is what "service-side deduplication" means. Where you do not control it, your only lever is the request contract — a stable key the service can recognise. Both routes end the same place, and both require the key to survive the model regenerating the request.
You do not, from the agent's side. In the measurement above, agents reported success in 90% of the episodes where they had duplicated an effect. The only reliable source is the service that holds the effect. Count committed effects per logical action, and treat the ratio of effects to intents as a first-class metric.
It reduces the frequency and does not remove the class. A ceiling of 14 days instead of 8 hours changes how often the restore happens; it does not change what happens to an in-flight action when it does. Longer ceilings also make the failure rarer and therefore less tested, which is its own risk.
The order matters. The measurements say the harness barely matters and the contract explains most of the variance, which is a strong hint that the useful work is in the contract between the agent and the services it writes to — idempotency keys on every write, a durable effects ledger outside rewindable state, and an explicit replay-or-fork decision at every restore. Before you rewrite anything, provoke the failure once. Kill the process in the window between the tool returning and the state committing, then count the effects on the service side. What you find will tell you which half of the agreement is missing.
If you want a disposable host to run that drill properly, the 3-day and 7-day trial plans start at $1.9, and the same crash-recovery discipline is worth applying to any always-on deployment — see why a 24/7 agent dies in week three for the slow accumulations that show up only after weeks of continuous running.
Related reading on this blog: at-least-once delivery and duplicate side effects, why a tool call with no deadline stops the whole loop, why your agent's log rotation is a durability setting, and the four sites where an agent's memory accumulates.
As digital transformation accelerates, businesses in th...
Port 25 is the default port used for sending emails thr...
Introduction: The "Cloud Tax" is Real For years, Ama...