Your AI Agent Retries Are Duplicating Side Effects, and At-Least-Once Is Why

Retries are what keep an automated workflow alive across a flaky network, and the same retries are what make it charge a card twice, because at-least-once delivery is the only guarantee any messaging system actually gives you.

This is not a model problem. An agent asked to issue a refund does not issue two refunds because it hallucinated. It issues two because the first call timed out after the refund had already been recorded, the framework classified the timeout as a failure, and the retry created a second one. Nothing in that sequence is a reasoning error. It is a coordination error, and it is the single most common way a production agent causes damage that no offline evaluation will ever catch.

The fix is old, boring and well specified: give every action that changes something outside the agent a stable key, store the first result, and make the second arrival replay that result instead of repeating the work. This article is about why that is harder than it sounds, what the current evidence actually supports, and where the responsibility has to live.

Five ways the same action happens twice

Duplicate side effects are usually described as a retry bug. Retries are only one of five entry points, and the other four survive even if you fix your retry policy.

1. A timeout after dispatch. The tool call reached the downstream service and the service completed the work. The response never made it back before the deadline. The caller cannot distinguish this from "never arrived", so it resends.
2. Transport-level retries you did not write. Load balancers and HTTP clients retry on 502, 503, 504 and connection resets by default. The first attempt frequently succeeded.
3. The scheduler fires at least once, and occasionally twice in one slot. A job that runs on a "job name plus hour bucket" lease does one pass per hour. A job that relies on the clock alone can run twice when a worker restarts mid-slot.
4. A crash lands between the action and the completion mark. The payment went through; the process died before writing "done". On recovery, the work looks unfinished because the record of finishing it never persisted.
5. A person re-runs the job. Someone sees a stalled task, hits run again, and the queued action fires a second time.

A published production write-up from a team running a fleet of agents lands on the same list, from the other direction: the model retries a failed tool call, the scheduler double-fires, two entry points post the same record, a crash lands after the action, and a human re-runs something that already ran. Five mechanisms, one symptom. Their fleet-level write-up was published on 10 October 2026 and describes what they now enforce, including two incidents they caused themselves: a double publish in August 2026 before a lock existed, and a routine that died holding a lock and blocked a daily slot in September 2026.

The useful conclusion is that "make the retry smarter" is the wrong layer. Four of the five mechanisms are outside your retry code. Whatever protects you has to be attached to the action, not to the attempt.

At-least-once is the only guarantee you get

It is worth reading how the largest messaging service states this, because the wording is blunt. Amazon's SQS documentation on at-least-once delivery explains that it stores copies of your messages on multiple servers for redundancy, and that on rare occasions one of the servers holding a copy is unavailable when you receive or delete a message. The copy is not deleted from the unavailable server, and you may receive that message again. The documentation then gives the instruction that matters: design your applications to be idempotent, because they should not be adversely affected by processing the same message more than once.

That is a platform explicitly telling you that delivery is not exactly-once and that handling duplicates is your job.

The same documentation offers the partial escape. A FIFO queue, unlike a standard queue, does not introduce duplicate messages if you retry a send inside its five-minute deduplication interval, either by explicitly supplying a deduplication ID or by enabling content-based deduplication, which hashes the message body and not the attributes. Note the shape of that guarantee. It is bounded to five minutes, it is scoped to the queue's own send path, and it says nothing about what happens after a consumer has already acted. It is a useful local rule, not a general answer.

Exactly-once processing, in other words, is not a delivery mode you can select. It is a property you construct at the point where the effect happens, and in most real systems that point is your code, not the queue.

The failure is an unverifiable outcome, not a failed call

The reason this is hard is that the interesting failure is neither success nor failure. An arXiv preprint submitted on 31 July 2026, "Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures" (arXiv:2608.02645), names the three behaviours that break a conventional retry policy. Existing agent frameworks, the authors observe, typically assume tool calls are atomic and return a binary success or failure signal. Real systems are not atomic: a call can time out after dispatch, a new record can become visible only after a delay, and a partial state update can leave the caller unsure what actually succeeded. Those mismatches, the paper says, produce duplicate actions, unnecessary tool executions and confused task success reporting.

Once you see the problem as "I cannot see the state", the design follows. A conventional retry policy answers one question: did I get a success signal? That question is unanswerable when a response can disappear after the action already happened. The paper's proposed wrapper shifts the question to: does the target system now show the intended result? That is a question you can answer, because you can look.

What "idempotent" has to mean for a side effect

The word gets used loosely, so here is a workable definition. An action is idempotent when running it twice has the same effect as running it once. For an agent that means three things have to be true at the same time.

• Every side effect carries a key that identifies the logical operation, not the network attempt.
• The first result is stored somewhere durable, keyed by that same key.
• A second arrival with that key returns the stored result instead of performing the action again.

Notice what is not in that list. It is not "the tool does not return an error when called twice". A double-charge that is silently accepted is still a double charge. Idempotency is a statement about effect, and the only way to make a statement about effect is to key the operation and remember what happened to it.

The window is a design decision, not a constant

Every implementation of this pattern has to answer the same question differently: how long does a key have to be remembered? Too short and a late retry, after a workflow resumes from a human pause, executes the action again. Too long and you are retaining response payloads that may contain personal data far beyond policy. The platforms that document this publicly do not agree on a number, which is the point.

Two details in that table are easy to miss and both matter. First, in every row the key comes from the caller. A key minted downstream, inside a retry helper, is regenerated on each attempt and therefore deduplicates nothing. Second, Stripe's behaviour of replaying the saved response even after a server error is the property people usually forget to build for themselves: if a retry returns a different body than the first call, the model has no way to know the work already happened, and it may try to "correct" a duplicate it just created.

Four layers, each answering a different question

A single lock or a single unique index is not a strategy. The pattern that holds up under all five repetition mechanisms stacks four controls, and each one answers a different question about the second arrival of the same work.

1. A stable natural key with an upsert. "Is this the same record?" The key must be stable for the underlying record, not for its current state, and the write is an update rather than an insert.
2. A lock on the slot, taken before the side effect. "Is another run already doing this job?" A refusal means stop. Releasing the lock only after the effect is durably recorded is what prevents a zombie run's late release from freeing a live retry's lock.
3. A ceiling on retries, with a stop code. "Have we tried this too often?" After a small fixed number of attempts the system should open exactly one task for a human and return a code the caller treats as permanent failure. A ceiling nobody reads is a log line, not a control.
4. Quarantine with a written reason. "Should this wait for a person?" Parked work records a machine-readable hold reason and goes to a named owner. A hold with no reason is a lost item.

The sequence matters. Layer 1 stops the most common duplicate by construction. Layer 2 stops the concurrent one. Layers 3 and 4 stop the ones that no amount of keying will fix, because the underlying action is genuinely ambiguous and a human has to look at it.

The trap: the same key with different arguments

There is one failure mode that active key-based designs get wrong, and it is worth stating separately because it is silent. The agent retries what it believes is the same step, but with mutated arguments, while reusing the old key. A different amount, a different recipient, a different target record. If the receiver only checks "have I seen this key", it will return the cached result of the previous, different operation and report success. The caller now believes a new action happened. It did not.

The fix is a payload hash stored beside the key. On reuse: same key and same hash returns the cached response; same key and a different hash is rejected with an explicit conflict error. Never overwrite silently. A spike in those conflicts is not noise, it is a signal that key minting or argument handling has a bug, and it deserves to page someone.

Where durable replay ends and the target system takes over

The largest open-source workflow engine now ships this as a first-class feature, and reading its documentation shows exactly where the limit is. Airflow's durable agent replay fingerprints each request and caches completed model and tool steps, so a task retry can return the stored result instead of re-running a step that already had a side effect. That closes the common case.

The gap it leaves open is precise, and the documentation is honest about it. If a tool completes its external action but fails before the result enters the cache, a later retry may execute the action again. Caching, in other words, protects the retry path but cannot protect the window between "the effect happened" and "we recorded that it happened". Closing that window is only possible in the target system, with an idempotency check or a uniqueness constraint on the write itself. This is why the four layers above are not optional extras around a caching system. They are the part the cache cannot cover.

What the evidence actually shows

It is worth being precise about how strong the case is, because the topic attracts confident claims. The 31 July 2026 preprint tested a verification-aware wrapper against a retry-only baseline in a controlled simulated environment with injected non-atomic failures, across several task templates. On a simulated invoice task, the retry-only baseline produced duplicate side effects in 32%, 52% and 76% of runs at low, medium and high fault levels. The wrapper reduced those to 0%, 16% and 20%. Component testing attributed most of the benefit to the verification step: checking external state before retrying lowered duplicate actions from roughly 42% to 20%.

Read that honestly and three things are true. The effect is large and it is largest exactly at low fault levels, where duplicates went to zero. Verification alone is doing most of the work, which is good news because it is the cheapest part to add. And the study is a preprint tested in a simulation, so it should inform engineering decisions rather than settle the question. Nobody has published an industry-wide fix for this, and anyone claiming otherwise is overstating a useful, narrow result.

A hint is not a control

There is a tempting shortcut: the Model Context Protocol already defines tool annotations, including idempotentHint, destructiveHint, readOnlyHint and openWorldHint. A tool that declares idempotentHint: true would seem to have answered the question for you.

It has not: those fields are declared as hints, and the specification's own guidance is that they should not be treated as faithful descriptions of tool behaviour from an untrusted server. A hint is a label the tool places on itself. It is not an enforced property of the exchange. The security-relevant taxonomy follows the same logic. The OWASP Top 10 for Agentic Applications 2026, published by the OWASP GenAI Security Project on 9 December 2025, lists tool misuse and cascading failures as separate risk classes, and a duplicated side effect can feed directly into either: an agent that repeats a destructive call is tool misuse in practice, and one action fanning out into several through retries is a cascade. In both cases the mitigating control lives on the side that performs the work, not on the side that describes it.

The practical rule is short. If the side doing the work does not enforce the key, you do not have idempotency. You have a comment.

Five questions to ask before an agent goes live

These are the questions that separate a design from a wish, and they are all answerable in writing.

1. Does every side effect carry a key that the caller chose, and is that key stable for the logical operation rather than the network attempt?
2. What happens when the same key arrives with different content? The correct answer is an explicit error, not a silent overwrite.
3. How long is the deduplication window, and is it longer than the longest realistic retry plus human-resume delay?
4. What is the stop code when the retry ceiling is hit, and which component reads it?
5. Where does a quarantined item go, and who is named as its owner?

If any of those answers is "the model handles it", the system is relying on reasoning to solve a state problem, which is the one thing reasoning is worst at.

When running the four layers yourself stops making sense

All four layers are runtime concerns, and none of them live in the model. They have to run somewhere that is up when the action fires, holds durable state across restarts, and survives the crash that lands between the effect and the completion mark. On a laptop that is a contradiction: the machine that runs the agent is also the machine whose sleep, Wi-Fi drop or closed lid produces the ambiguous outcome in the first place. The gap between "the effect happened" and "we recorded it" is exactly the gap a personal device cannot close.

This is the point at which hosting the agent stops being a personal-infrastructure decision and becomes part of the correctness argument. A continuously running instance with a stable public network path turns the crash window from minutes into a routine restate, and it is what the OpenClaw deployment image on SurferCloud is built for: a preconfigured CentOS Stream 9 image in Japan, Singapore and US data centres, launched from the console and connected to a model provider in about three minutes, with no build step to reproduce on each restart. Three plan shapes cover the range the pattern needs, from 1 vCPU / 2 GB up to 4 vCPU / 8 GB, so the runtime that holds your keys and locks can be sized for the fleet rather than the demo.

To be clear about what that does and does not do for you: hosting the runtime closes the availability and crash-recovery half of the problem, because the instance that stores your keys and locks stays up. It does not exempt you from the first half. The keys, the payload hashes, the retry ceiling and the quarantine reason are still yours to design, and a server that is always on will faithfully execute a duplicate just as fast as it executes a correct action.

The four layers are what make the retry safe. Somewhere that never sleeps is what makes them possible. The OpenClaw image on SurferCloud covers the second, with the deployment steps, plan specifications and current pricing on the OpenClaw cloud deployment page, and the underlying instance options documented under lightweight cloud server plans.

Frequently asked questions

Is idempotency the same as exactly-once delivery? No. Exactly-once delivery is not something a queue can promise you, and at-least-once is the honest guarantee. Idempotency is how you turn at-least-once delivery into at-most-once effects at the point where the action actually happens.

Can the agent framework not just handle this for me? Framework-level caching covers the common retry path, and it is worth using. It cannot cover the window between a completed side effect and the record of that completion, which has to be closed in the target system.

Do I need a key on read-only tools? No. Classify tools as read, side-effecting, or unsafe. Require a key for the second and third categories, and require human confirmation for the third.

What is the right deduplication window? Long enough to cover your longest realistic retry plus any human-resume delay, and short enough that you are not retaining sensitive response payloads beyond policy. Twenty-four hours is a common default for money-moving actions; minutes are often enough for chatty read-like tools.

Does a tool annotation solve this? No. Tool annotations are declared hints about behaviour. The side that performs the work has to enforce the rule.

Get the ordering wrong and the failure is quiet: the agent looks correct, the logs look clean, and the only evidence is a duplicate that a human notices days later. Get it right and retries become what they were always supposed to be, which is boring recovery.