An Agent's Memory Leak Has Four Sites, and Your Log Line Sees One

Your agent's memory metric is answering a question you did not ask. You wired up a log line for heap used, you watched it stay flat for six weeks, and the host still ran out of memory. The line was not lying. It was reporting the smallest of four places where a long-running agent accumulates memory, and it is the only one that has any obligation to report itself.

This matters because the fix for each site is different, and the wrong fix is expensive. Adding RAM absorbs a leak instead of removing it. Restarting on a schedule converts an unbounded failure into a clock. Raising a limit the runtime already ignores changes nothing. Before you can choose, you have to know which of the four sites is growing, and that means knowing what each number your runtime prints actually covers.

Four Accumulation Sites, and Which Line Sees Each

The Node.js runtime exposes exactly five memory fields, and their definitions are worth reading once rather than assuming. Resident set size is the space the process occupies in main memory, and it includes the heap, the code segment, and the stack. Heap total and heap used refer to the V8 heap. External is the memory used by C++ objects bound to JavaScript objects that V8 manages. Array buffers is the memory allocated for ArrayBuffer and SharedArrayBuffer, including every Node.js Buffer, and it is a subset of external. That is the whole list.

Read that list as a coverage map rather than a health report, and the shape of the problem appears immediately.

Two of the four are invisible to every number the process prints about itself. One is visible only through a number nobody alerts on. Only the first is instrumented by default, and it is the one that behaves best.

Site One: The Heap, the Only One That Reports Itself

Managed runtimes make their heap ceiling explicit, and the ceiling is usually generous enough that you never meet it, which is exactly why the failure is confusing when it arrives. The host can show free memory while the process has already hit its own internal limit. Node.js will tell you the number it is enforcing if you ask it directly.

The useful habit is not the ceiling, though. It is the shape of the curve. Logging heap used on an interval costs nothing and answers a question no dashboard will: a sawtooth that returns to a consistent baseline is healthy collection, while a baseline that creeps upward every cycle is retention, and no amount of collector tuning will fix retention, because the baseline is telling you that something is being held that should have been released.

This site is the best understood of the four, and it is already the subject of a longer treatment of the three slow accumulations that kill a persistent agent. What matters here is the part that is usually skipped: when the heap line is flat, most engineers conclude there is no leak. What they have actually established is that this particular site is not the one growing.

Site Two: Buffers, the Memory the Runtime Accounts For and Your Code Owns

Node.js gives you a second, mostly unremarked accounting line. Array buffers is tracked, it is a subset of external, and in an agent it grows for a reason that has nothing to do with a coding error.

Consider what an agent moves through memory in an ordinary hour. A tool call returns a JSON body. A document summarizer loads a PDF. An OCR step holds a decoded image. A batch job reads a spreadsheet. A model response arrives as a large string and is immediately re-encoded. Every one of these is an ArrayBuffer, and every one of them is subject to garbage collection in principle. The retention is not accidental. It is caused by the pattern that makes an agent useful: the same identifier gets asked about twice, and a cache is the obvious answer.

Caching a tool result keyed by a request hash that varies with a timestamp is a cache with a zero hit rate and an unbounded key space. It is also invisible, because the entries are reachable, correctly referenced, and therefore not leaks by any definition a heap profiler will give you. The memory is accounted for. The growth is designed.

There is a second variant at this site that is genuinely a defect and worth separating out. When a tool call times out, the client-side timeout reclaims the waiter, not the downstream work, so a response that arrives after the deadline can still be retained by a handler that no longer has a consumer. Retries multiply the effect, because each attempt allocates its own response buffer while the abandoned ones remain referenced, which is the same mechanism that makes naive retry logic duplicate side effects on the receiving end. Neither of these produces a heap leak, and both show up as a rising array-buffers line, which makes this the one off-heap site you can actually measure from inside the process.

Site Three: Allocator Retention, and Why RSS Climbs in Steps

Below the runtime sits the C library allocator, and its documented behavior is the reason a flat heap line and a rising resident set size can both be true for months.

The glibc manual is direct about the asymmetry between the two allocation paths. Memory obtained with mmap has the significant advantage that the allocated blocks can always be independently released back to the system. The heap, by contrast, can be trimmed only if memory is freed at the top end. Freed space in the middle of the heap is not returned; it goes onto a free list for reuse.

Three documented parameters decide how much is retained.

Trim threshold specifies the minimum size of the contiguous free region at the top of the heap before the allocator releases it back to the system, and its default is 128 KiB. Top pad defines the padding applied when the program break moves, and it also fixes how much free space is deliberately preserved at the top of the heap during a trim; its default is 128 KiB as well. Setting the trim threshold to minus one disables trimming completely. The manual does not present these as tunables with obvious upsides. It presents them as a trade between making more system calls and wasting unused memory at the top of the heap.

The reason this matters more than the defaults suggest is the dynamic threshold, which is documented but easy to miss. The allocator starts with an mmap threshold of 128 KiB and raises it over the life of the process: when a block larger than the current threshold, and no larger than the platform maximum, is freed, the threshold is adjusted upward to the size of that freed block. The manual then states the consequence in one sentence that explains most step-shaped RSS graphs. While dynamic mmap thresholding is in effect, the threshold for trimming the heap is also dynamically adjusted, to twice the dynamic mmap threshold.

Worked through, the effect is this. A benchmark on 64-bit Linux where the largest freed block reaches the platform maximum of 32 MiB leaves a trim threshold of 64 MiB. The process can hold up to 64 MiB of contiguous free memory at the top of its heap and still not return a single page, because the threshold that governs the return has grown to match the workload's largest freed allocation rather than the workload's current need. On a two-gigabyte plan, that is three percent of the host sitting in a free list that only reports itself as resident.

A second mechanism compounds it. The arena count is not fixed. The default limit is determined by an arena test parameter whose documented default is eight on 64-bit systems, and arenas are created on demand as threads contend. The manual is explicit about the trade: more arenas means lower per-thread contention and higher memory usage. What it does not need to say is that the reverse never happens. Arenas are not dissolved when contention subsides, so the resident set ratchets to the high-water mark of the busiest hour the process has ever had and stays there. Allocator fragmentation adds the last increment, since the fastbin mechanism deliberately does not merge adjacent free blocks, and the manual notes that this can increase both fragmentation and the program's overall memory footprint.

So the step function has a mechanism. Steps align with bursts, not with time. Each burst of concurrent work buys arenas and raises the mmap threshold; the memory stays resident; the heap line stays flat because none of this is heap. Two rules follow. A step that is followed by a long plateau is retention, and will not resolve itself. A gradual slope is different, and usually means something is genuinely accumulating keys.

One trap belongs at the end of this section because it converts a diagnostic into a permanent condition. Dynamic adjustment of the mmap threshold is disabled if any of the four relevant parameters is set, whether in code or by environment variable. Pinning the mmap threshold to a tidy value is a common piece of advice, and it freezes the trim threshold at a static number for the life of the process.

Site Four: The State Store, the Ceiling That Is Not Process Memory

The fourth site is specific to agents, and it is the one that will retire your deployment if you do not plan for it. A process that runs for months does not keep its working state in memory. It keeps it in a transcript, a checkpoint table, an embedding index, or all three, and that store grows monotonically with the number of sessions, because every session contributes and none of them are deleted by default.

Until recently the cost of managing this was asserted rather than measured. A benchmark submitted on 22 October 2026 makes the accounting concrete. It pairs full-history and summary-only continuations from a shared repository state and measures what a specific history replacement actually destroys, covering 35 repairs, 146 agentic sessions, and 106 production chains across 14 summarizer backbones.

The headline result is that stable damage from compaction ranges from 5 to 64 percent of the 22 full-history successes, depending only on which model does the summarizing. But the number that should change your design is in the retention measurement. Language-model summaries retained at most 16.0 percent of recurring details, against 79.1 percent for mechanical extraction, and production summaries retained 15.7 percent at the first boundary. Compaction buys you a smaller store and charges you most of the detail that made the store useful.

A second paper submitted the following day, 23 October 2026, extends the same argument to the runtime around the model. Across 111 evaluation settings and 52,168 trajectories it shows that harness rankings vary by model and by benchmark, and that similar task scores can accompany entirely different execution profiles. One of its five measures exists purely to count tokens consumed by execution, explicitly including failed attempts, which is the same insight from a different direction: the cost of a persistent process is not the cost of its successes.

There is a security consequence to this site that has nothing to do with bytes. Everything a long-running agent can do, it can do because its credential list is the boundary of what it can reach, and a store that never deletes is a store that never narrows that boundary.

Together these two results describe the real ceiling on a long-running agent. Compaction discards the state that finishes tasks, mechanical extraction preserves details but does not scale indefinitely, and every retry and every failed attempt is charged to the same store. The store is on disk, so it never appears in any memory graph, and it is the constraint that sets your instance size.

Five Things That Look Like a Leak and Are Not

Confident misdiagnosis is worse than no diagnosis, because it produces a fix that consumes a maintenance window and changes nothing. Five patterns account for most of the false positives on this class of workload.

The fourth row deserves one clarification, because it is where process accounting is most often misread. Page cache in general is not charged to your process, but resident file-backed pages the process has mapped are, and they are reclaimable. A process holding a large mapped data file will show a resident set that is inflated by pages it does not own and can release on demand. This is memory doing useful work, and treating it as a leak leads to a cache that is needlessly disabled.

A Ten-Minute Test That Names the Region

The goal is not to find the leak. It is to eliminate three of the four sites so that the remaining search is bounded. Four measurements are enough.

Two details turn this from a list into a diagnosis. First, run the test on a host rather than in a container whose limit you cannot read, because the cgroup counters are the only place the difference between a runtime limit and a host limit is written down. Second, take the measurements on a schedule and write them to a log rather than a dashboard. The graph you want is not the one you look at; it is the one you can grep at the moment something breaks.

One mechanism deserves a name here because it explains a case that looks like a leak and is a control-plane problem. When a process exceeds its memory limit, the kernel does not kill immediately on breach. It attempts reclaim first, and only invokes the out-of-memory killer inside that group when the usage cannot be reduced. An increase in the counter that records near-breach events without a matching increase in kill events is therefore not reassurance. It is a group that is being repeatedly bailed out by reclaiming its own file pages, and it is one allocation away from a kill the next time those pages are dirty.

Note also which limit you are comparing against. A soft limit throttles the workload under heavy reclaim pressure and never invokes the killer, while a hard limit does. A group sitting at its soft limit for hours is being slowed continuously rather than failing once, and that pattern is usually reported as intermittent latency rather than as a memory problem, which is why it survives so long.

Do Not Absorb a Leak With Memory You Already Bought

Once the region is known, the tempting fix is to buy headroom. It is the wrong fix at three of the four sites, and the reasoning is worth stating plainly because the cost is not obvious on the invoice.

At site one and site two, retention is proportional to the number of distinct keys your workload sees. Memory you add does not change the coefficient; it changes the time. The failure you had at three weeks you will now have at eleven weeks, with the addition that the restart now happens during a period of higher traffic, because that is when the key space grows fastest. You have paid for a delay and a worse crash.

At site three, adding memory interacts badly with the ratchet. If the resident set rises to the high-water mark of the busiest hour, then giving the process more memory raises the ceiling that the high-water mark can reach, so the new headroom is not linear: it is consumed by bursts that previously would have been capped. The allocator is behaving exactly as documented, and the extra memory is being spent on a high-water mark rather than on throughput.

Site four is the only place where buying memory can be correct, and even there the useful purchase is not on the agent host. It is on the store, which is why the sizing question for a long-running agent is usually a storage and retention question wearing a memory costume.

What works instead is a ceiling that the process cannot negotiate with, plus a policy that fails early. A hard limit below the host's capacity turns a slow uncontrolled rise into a clean restart of one process. A log line per interval containing memory usage, restart count, and completed tasks costs nothing and answers most retrospective questions without a profiler. And a retention policy on the state store is the only item on this list that removes a constraint rather than managing it.

Where a Managed Host Changes the Arithmetic

There is a structural problem with everything above, and it is not a technical one. The measurements that eliminate three of the four sites are host-level measurements. Resident set size, the memory map, the kernel log, the cgroup counters, and the difference between a soft and a hard limit all live outside the process, and the process is the thing being killed. An agent that is asked to report on its own memory exhaustion is reporting from inside the failure it is describing.

That is the constraint a managed host is actually built for, and it is the point at which running the agent on your own hardware stops making sense. Not because the hardware is too slow, but because the observability you need is on the wrong side of the process boundary, and because a ceiling enforced by the platform is enforced whether or not the application cooperates with it.

For a persistent agent the requirements reduce to a short list. The host must outlive your laptop. The deployment must not require you to rebuild the environment. The instance must be big enough to hold a high-water mark without a kill, and small enough that a runaway process is stopped rather than tolerated. And the platform must expose the memory ceiling rather than leaving it implicit. The OpenClaw hosting plans are the concrete instance of that list for this workload.

A Sizing Note for a Process That Never Returns

SurferCloud provisions this workload from a prebuilt OpenClaw image, which removes the environment work rather than the operational work, and it is worth being specific about what the plans currently carry, because the numbers decide whether the high-water mark fits.

Two notes qualify the table, and both matter for the argument in this article. The first quarter plan at $4.66 per month carries the least memory of the three at 2 GB, which is precisely the configuration where an allocator high-water mark is most likely to reach the ceiling, so the cheaper plan is the one that needs the retention policy most. The monthly plan at $6.75 differs in bandwidth rather than in memory: 30 Mbps peak on shared bandwidth, against dedicated bandwidth on the other two, so it is the one to pick when the agent's outbound calls matter more than its resident footprint. Neither the monthly price nor the quarter price is a first-month rate, which matters because a step-shaped memory graph takes longer than a month to distinguish itself from a slow slope.

The page states a configuration range from 1 vCPU and 2 GB up to 4 vCPU and 8 GB, deployable in about three minutes with a prebuilt image, connected to leading model providers including GPT-5, GPT-4, DeepSeek, Claude, Gemini, and GLM through an OpenAI-compatible API or a custom endpoint. Nothing on the page promises memory controls the platform does not expose, so treat the instance size as the memory control and set your retention policy accordingly.

FAQ

Is a slow rise in resident memory always a leak? No. Two of the five patterns in the table above are reusable memory rather than retained memory. Reclaimable mapped file pages and allocator free space both count as resident and both can be recovered. What separates them from retention is the shape: retention slopes or steps upward under steady traffic, while reusable memory correlates with write volume and recovers on its own.

Will restarting on a schedule fix it? It will reset the resident set, which is why it feels like a fix. It does not change the coefficient, so the process will reach the same ceiling again, and the restart will land at a time chosen by the retention rate rather than by you. Scheduled restarts are a reasonable stopgap while you narrow the region, and a poor permanent answer at sites one, two, and three.

Should I raise the runtime heap ceiling instead? Only after confirming the plan has memory to back it. The flag that raises the limit sets a limit; it does not create memory. Raising it on a host without headroom converts a clean out-of-memory exit inside the runtime into a system-wide problem, because the runtime stops protecting the process before the kernel starts protecting the host.

Does context compaction release memory? It reduces the store, not the process. That is the right lever for site four, and the benchmark above shows what it costs: at most 16.0 percent of recurring details retained by model-generated summaries, against 79.1 percent for mechanical extraction. Treat compaction as a retention policy with a measured price, not as a free reduction.

Summary

Instrument the smallest line and you will learn the least. An agent accumulates memory in four places, and two of them are invisible to every number the process prints about itself.

The heap reports itself and behaves best. Buffers are accounted for and grow by design. The allocator retains what it freed and shows up only in the resident set. The state store sets the real ceiling while never appearing in a memory graph.

Measure outside the process, because the process is what gets killed. Then, and only then, decide whether to fix the retention or buy the headroom.