SurferCloud Blog SurferCloud Blog
  • HOME
  • NEWS
    • Latest Events
    • Product Updates
    • Service announcement
  • TUTORIAL
  • COMPARISONS
  • INDUSTRY INFORMATION
  • Telegram Group
  • English
    • 中文 (中国)
    • English
SurferCloud Blog SurferCloud Blog
SurferCloud Blog SurferCloud Blog
  • HOME
  • NEWS
    • Latest Events
    • Product Updates
    • Service announcement
  • TUTORIAL
  • COMPARISONS
  • INDUSTRY INFORMATION
  • Telegram Group
  • English
    • 中文 (中国)
    • English
  • banner shape
  • banner shape
  • banner shape
  • banner shape
  • plus icon
  • plus icon

Your Agent's Checkpoint Is a Position, Not a Receipt

November 16, 2026
19 minutes
INDUSTRY INFORMATION,TUTORIAL
37 Views

A checkpoint records where an agent was, not what it did. Restore one after a crash and you get the position back, not the world. The payment that cleared, the ticket that was filed, the message that was sent, the row that was committed — every one of those happened in the seconds between the last checkpoint and the failure. The agent wakes up with a stale copy of its own memory and no idea any of it took place, so it does the reasonable thing: it does them again.

This is not a bug in any particular framework. It is the arithmetic of two independent commit paths that were never designed to agree. Your agent commits its state on one schedule. The services it writes to commit their effects on another. Recovery asks one of them to stand in for the other, and it cannot.

The failure surfaces as a duplicate. A second charge. A second deployment. The same webhook delivered to a customer twice. The agent reports success, because from its point of view it succeeded. The accounting is off by exactly the work that fell inside the window.

Why restoring state feels like recovery

The reason this is hard to see is that checkpoint-restore works beautifully in a development environment. You kill the process, you restart it, the graph resumes from the last super-step, and the run completes. The demo passes. Nothing in it touches an external system, so there is no second world to disagree with.

Every durable-execution system you can read about starts from the same premise. Temporal's documentation describes durable execution as the ability of a workflow to maintain state and progress through failures, achieved by an event history that records the state of the execution at each step, so that the run can resume from the last recorded event. LangGraph's persistence layer describes checkpointers as persisting a thread's graph state as checkpoints, with a separate store for application data that lives outside the graph state. Both are accurate descriptions of what the mechanism does. Neither claims to describe what happened in the world.

Read the words carefully. The event history records the state of the execution. The checkpointer persists graph state. State is the agent's belief. It is a record of what the agent knew, not a record of what it caused. A resume-from-last-event is a resume of a belief, and beliefs can be wrong in exactly one direction that matters: they can be behind reality.

The gap is one commit wide

Put a timeline on it. A tool call leaves the agent, arrives at a service, and the service commits an effect. The agent then records that the call returned. Two writes, to two different systems, at two different moments, and the failure can land between them.

  • The tool commits the effect, the agent crashes before recording the result. On restart the agent sees a pending action and re-issues it. This is the duplicate case.
  • The tool never commits, the agent crashes. On restart the agent re-issues it correctly. This is the case retries are designed for, and it is the only one that tests exercise.
  • The tool commits twice, because the request was delivered twice at the transport layer. The agent records one result. This is the case where no amount of agent-side logic helps, because the agent only ever saw one acknowledgement.

Only the second case has a clean answer. The first and third are the same failure seen from two ends: an effect exists in the world whose existence the agent's state does not reflect, and the agent's state cannot be repaired by replaying its own history, because that history never contained the fact in the first place.

There is a related trap that follows from the same gap. A checkpoint can restore the agent's state while it is still behind the world, and then the agent does not replay the same request — it generates a new one. A model asked to redo a step does not reproduce its previous tokens; it produces a semantically similar request with different wording, a different idempotency token, a different amount. A service that keys deduplication on the exact request body sees a new request and executes it. The March 2026 preprint on this, which introduced the term semantic rollback attacks, is an early warning rather than settled evidence, but its mechanism needs no statistical support to be true: if the retry is regenerated rather than replayed, the request that arrives is not the request that landed.

Three questions a checkpoint cannot answer

Before any retry decision, an agent has to establish one of three outcome states for the action it is about to repeat. The checkpoint tells it none of them.

Question What the checkpoint says What the service knows
Did the write commit? Nothing. The call was in flight, so no result was recorded. Yes or no, but only if you ask it in the right way.
How many times did it commit? Nothing. Nothing was acknowledged, or one acknowledgement arrived. One, twice, or partially, depending on the transport.
Was it the same request? Nothing, once the agent regenerates it. Only if the request carried a stable key.
The three outcome states a recovery path must resolve, and the one system that cannot resolve them.

Notice where the answers live. Two of the three are only knowable by the service, and the third is only knowable if the request was designed to be recognisable. This is why the fix is not a better checkpoint. It is a contract between the agent and the services it writes to, and the checkpoint sits outside it.

What the measurements actually show

This is no longer a hypothetical. A September 2026 study built a deterministic sandbox of six services with realistic contracts — optional idempotency keys, eventually consistent reads, and deliberately missing read paths — injected twelve fault modes at the service boundary, including late commits, redelivery, and partial batches, and graded every episode against a ledger of effects that had actually committed. Across 25,930 episodes, nine recent models, three production agent harnesses, two contract variants, and fifteen recovery conditions, the results separate cleanly by whether the agent could find out what happened.

Condition Duplicate rate What predicts the outcome
A read-back immediately reveals the result 0.5% for frontier models instructed to act exactly once The model. It explains 53% of the explained variance; weaker models duplicate far more often.
The request is still in flight 56% for the same frontier models The contract. It explains 81% of the explained variance.
The transport delivered the request twice 74% for the same frontier models The contract, again. The agent never saw the second copy.
Duplicate side-effect rates from the LIMBO sandbox, 25,930 episodes (arXiv:2609.29095, submitted 24 September 2026). Source: the paper's own figures.

Three findings in that set are worth more than the headline rates.

The first is that no verification-only policy can be exactly-once under late commits without a bound on how long a request may remain in flight. Waiting seems like the safe answer, and it works when the in-flight delay is short and known. But with heavy-tailed delays, even an hour of waiting per episode falls short of simply attaching an idempotency key to every write.

The second is the size of that difference. Offering an idempotency key on every write lowered the duplicate rate from 28% to 4%, in the paper's own comparison, because agents actually use keys when they exist. That is a seven-fold reduction from a contract change, not a model change.

The third is the one that should change how you monitor. The harness barely mattered — a guard that attaches keys transferred across harnesses unchanged. And the agents reported success in 90% of the episodes in which they had duplicated an effect. Your success metric is not merely blind to this failure; it is systematically wrong about it, because the agent's own report is generated from the same stale state that caused the duplicate.

Replay is a decision, not a consequence of restore

The instinct when a checkpoint restores is to let the run continue from where it stopped. That is the wrong default, because it silently chooses replay over fork for every action in flight, and replay is only correct for the actions that never committed.

An October 2026 study formalises the test. It injects a controlled error, then replays the same downstream actions from the recovered state and from the target state, and requires that identical downstream actions produce identical outcomes — the paper calls this transition-consistent recovery. If the two runs diverge, the agent did not recover, whatever the final score says. Its finding is that current computer-use agents fail to exploit the recovery capability they already have, and that accurate model-based recovery produces measurable end-to-end gains. The practical version for anyone running agents in production is smaller and sharper: recovery is a state you can check, not a success rate you can infer.

So make the branch explicit. After a restore, the recovery path should decide, per pending action, whether to replay it, skip it, or escalate — and that decision should be recorded. The mechanism that enforces it is a replay-or-fork choice at the point of restore, not an automatic replay of every tool call in the pending list.

Why an agent's replay is worse than a program's replay

A deterministic program that replays an operation produces the same operation. The request bytes are identical, the key is identical, and a receiving service that has seen it before can recognise it.

An agent does not replay. It re-plans. Given the same restored state and the same goal, a language model regenerates the step, and the regenerated step is not byte-identical to the original. Sometimes that is harmless, because the action is idempotent by nature — a read, a lookup, a status check. Sometimes it is the entire problem, because the action is a write, and the service has no way to know that this new-looking request is a repeat of one it already committed.

This is why idempotency keys matter more for agents than for ordinary services, and why the keys have to be stable across the regeneration. A key derived from the request body fails the moment the model rewords the request. A key derived from the intent — the action type plus the logical target plus the task — survives the reword, because it was never a function of the wording.

Four scheduling facts that guarantee this happens to you

None of this would matter if crashes were rare. They are not, and the reason is that modern agent hosting is built around ceilings the agent does not control. These are not edge cases; they are defaults.

Layer Default Consequence for an in-flight action
Managed agent runtime Idle session timeout 900 seconds (15 minutes); maximum session lifetime 28800 seconds (8 hours) for microVM runtimes, up to 1209600 seconds (14 days) for capacity-provider runtimes The instance is terminated at the ceiling. Termination itself can take up to 15 seconds for logging and process completion. The session can continue on a newly provisioned instance — which is exactly the restore.
Container orchestrator Graceful termination default 30 seconds; then SIGTERM to PID 1, then SIGKILL to whatever remains One tool call that takes 60 seconds overruns the window twice over. If the kubelet or the runtime restarts while waiting, the cluster retries from the start, including the full original grace period.
Service manager Restart defaults to no; restart delay defaults to 100 ms; start rate limiting defaults to 5 starts within 10 seconds A framework with no restart policy simply stops. With one, a crash loop is capped: more than 5 starts in 10 seconds and the manager stops trying, and only a later manual or timer start, or a reset of the failed state, reactivates the restart logic.
Process signal SIGKILL cannot be caught, blocked, or handled No shutdown hook runs. Anything that was going to be recorded before exit is not recorded.
Recovery is a scheduled event, not an accident. Defaults are from the AWS AgentCore lifecycle documentation, the Kubernetes pod termination documentation, and the systemd service and manager documentation.

Two of these deserve emphasis. The start rate limit is the one people miss, because it turns recovery into a finite resource: a service that crashes and restarts every two seconds will exhaust its five-start allowance inside ten seconds and then stay down. And the 8-hour ceiling means that for long tasks the restore is not a rare event at all — it is part of the normal operating cycle.

An eight-hour ceiling against a twelve-hour task

Here is the arithmetic that makes the whole problem routine rather than theoretical. Current frontier models complete a fifty-percent-reliability task of roughly twelve hours, according to the 2026 METR time-horizon measurements as reproduced in the September 2026 State of Open Source AI report, which also notes that under eight hours models of both open and closed classes can generally do the job, and past twelve neither can. The default maximum session lifetime for a microVM-class managed agent runtime is eight hours.

Twelve divided by eight is one and a half. A task at the edge of what the model can do does not fit inside a single session, by design, and the boundary is enforced whether or not the agent is anywhere near finished. Every crossing is a restore, and every restore is a replay decision for whatever was in flight.

Run the same numbers at scale. A single long-lived agent on a managed runtime that hits its ceiling every eight hours is restored 90 times a month and 1,095 times a year. At each of those restores, any action that was mid-flight is a candidate for either a lost effect or a duplicate one. This is not a tail risk. It is the median path of any agent that runs continuously, which is precisely what a 24/7 agent is. The related failure modes compound here rather than in isolation: a service that dies in week three of continuous operation, or a memory footprint that grows across restores, will also be restarting on this schedule.

Build the recovery ledger, not just the checkpoint

The correction is a change of architecture, not of library. Keep the checkpoint for what it does well — restoring the agent's position, its plan, its conversation, its accumulated context. And put the record of effects somewhere the agent's own state cannot rewrite it.

  • Attach a stable idempotency key to every write, derived from intent rather than wording. This is the change that measured as the seven-fold reduction, and it works across harnesses unchanged. A key that survives the model regenerating the request is the whole point.
  • Keep an effects ledger outside rewindable state. A durable record of which actions committed, written by the layer that made the call rather than by the agent, is not subject to the same rollback as conversational state.
  • Check before you retry, and treat "unknown" as a distinct answer. An immediate read-back resolves the case cleanly. When the request is still in flight and no bound on that time exists, the honest answer is that the outcome cannot be established — and that is a case for escalating, not for guessing.
  • Bound in-flight time, or accept that waiting cannot substitute for keys. A known, short maximum delay makes waiting viable. A heavy tail does not.
  • Make the restore decision explicit and logged. Replay, skip, or escalate, per pending action, recorded. A restore that silently replays everything is a decision made by omission.
  • Stop trusting the agent's own success report for this failure class. In 90% of the duplicate episodes in the measurement above, the agent reported success. Measure duplicates per 1,000 writes as its own indicator, not as a footnote on the success rate.

A ten-minute crash drill

You can test the window in the time it takes to make coffee, and it is the one test most retry suites skip. Ordinary retry tests inject the failure before the tool runs. The interesting window is after the tool runs and before the state commit, and it needs to be provoked deliberately.

  1. Pick one write action with an external effect that you can observe directly and reverse if needed.
  2. Discard the agent's own success report for this test. Establish the effect's truth from the service side only — re-read the record; do not read the response.
  3. Send SIGKILL to the agent process in the window between the tool returning and the state commit. SIGKILL is the right signal precisely because it is uncatchable and no shutdown hook can cover for you.
  4. Restart the agent and let it recover from its checkpoint.
  5. Count the committed effects on the service side. One is a pass. Two is the duplicate the whole design is meant to prevent.
  6. Repeat with an in-flight delay inserted before the commit, so the tool is still running when the process dies. That is the case where the outcome is genuinely unknown, and where the recovery path must escalate rather than retry.
  7. Repeat with the request delivered twice at the transport layer. That is the case where only a key can save you, because the agent never sees the second copy.

If the first count is one and the second two, you have found the gap. It will not appear in any test that only fails before the tool runs, and it will not appear in any monitoring built on the agent's success rate.

Where this has to run

The recovery discipline above assumes something about the host: that the process is the agent's own, that you can send it a signal, watch it die, restart it on the same disk, and inspect what survived. You cannot run a ten-minute crash drill against a laptop that suspends, or against an environment where the process is managed by someone else's scheduler with rules you cannot see. The drill is only meaningful on a host you control, where a killed process leaves its checkpoint and its ledger exactly where the next start will look for them.

That is a modest requirement, and it is worth satisfying before the architecture rather than after it. A short-lived trial instance is enough room for the drill: a real environment with the same image, the same disk behaviour, and the same ability to be killed and restarted that production has. SurferCloud's UHost trial plan gives a 3-day or 7-day instance — 2 vCPU and 2 GB from $1.9 for 3 days or $3.9 for 7 days, on the same console and API as a full cloud server — which is exactly the kind of disposable host where you can kill the process in the dangerous window on purpose, as many times as it takes to see the duplicate. Full pricing is on the UHost trial plan page, and the same drill applies unchanged to a longer-lived instance.

The wider point is that crash recovery is not a property your framework can give you, because it is not a property of one system. It is a property of the agreement between the agent and everything it writes to. The checkpoint holds one half of that agreement, and the half it holds is the half that is allowed to be wrong.

Frequently asked questions

Does a checkpointer like LangGraph's make an agent crash-safe?

It makes the agent's state recoverable, which is a different claim. The documentation for these layers is careful about this: a checkpointer persists a thread's graph state, and a separate store holds application data outside the graph state. Neither is a record of what the agent caused in other systems. Crash-safety for side effects comes from the contract with those systems, not from the checkpointer.

Is waiting for the in-flight request to finish a safe recovery strategy?

It is safe when the maximum in-flight delay is short and known. The measurement above shows that with heavy-tailed delays, waiting cannot substitute for keys — even an hour of waiting per episode fell short in that comparison. If you cannot bound the delay, bound it or key the write.

Why not just make every tool call idempotent on the service side?

That is the strongest fix where you control the service, and it is what "service-side deduplication" means. Where you do not control it, your only lever is the request contract — a stable key the service can recognise. Both routes end the same place, and both require the key to survive the model regenerating the request.

How do I know a duplicate happened if the agent reports success?

You do not, from the agent's side. In the measurement above, agents reported success in 90% of the episodes where they had duplicated an effect. The only reliable source is the service that holds the effect. Count committed effects per logical action, and treat the ratio of effects to intents as a first-class metric.

Does a longer session lifetime remove the problem?

It reduces the frequency and does not remove the class. A ceiling of 14 days instead of 8 hours changes how often the restore happens; it does not change what happens to an in-flight action when it does. Longer ceilings also make the failure rarer and therefore less tested, which is its own risk.

Start with the drill, not the framework

The order matters. The measurements say the harness barely matters and the contract explains most of the variance, which is a strong hint that the useful work is in the contract between the agent and the services it writes to — idempotency keys on every write, a durable effects ledger outside rewindable state, and an explicit replay-or-fork decision at every restore. Before you rewrite anything, provoke the failure once. Kill the process in the window between the tool returning and the state committing, then count the effects on the service side. What you find will tell you which half of the agreement is missing.

If you want a disposable host to run that drill properly, the 3-day and 7-day trial plans start at $1.9, and the same crash-recovery discipline is worth applying to any always-on deployment — see why a 24/7 agent dies in week three for the slow accumulations that show up only after weeks of continuous running.

Related reading on this blog: at-least-once delivery and duplicate side effects, why a tool call with no deadline stops the whole loop, why your agent's log rotation is a durability setting, and the four sites where an agent's memory accumulates.

Tags : AI Agent ai agent hosting Cloud Server run ai agents 24/7 SurferCloud SurferCloud Promotion

Related Post

4 minutes INDUSTRY INFORMATION

Why Taipei Cloud Servers Are the Ultimate Sol

As digital transformation accelerates, businesses in th...

3 minutes TUTORIAL

What is Port 25 and Why Do Some Cloud Provide

Port 25 is the default port used for sending emails thr...

2 minutes INDUSTRY INFORMATION

Moving from AWS Lightsail to ULightHost: A Co

Introduction: The "Cloud Tax" is Real For years, Ama...

3-Day & 7-Day Trial at $1.9

GPU Special Offers

RTX40 & P40 GPU Server

Light Server promotion:

ulhost

Cloud Server promotion:

Affordable CDN

ucdn

2025 Special Offers

annual vps

Copyright © 2024 SurferCloud All Rights Reserved. Terms of Service. Sitemap.