Your Agent's Log Rotation Is a Durability Setting

An agent's trace log is not a debugging aid — it is the copy of record, and the rotation policy attached to it decides how far back the system can reconstruct what actually happened after a crash. This is not a metaphor and it is not a storage-hygiene preference. When a long-running agent dies at hour nineteen of a task, the only thing that can tell you which of two hundred tool calls already had a side effect, which approvals were granted, and where the plan diverged is the record the process wrote as it ran. If rotation deleted that record ten hours ago, the recovery procedure is not "resume" — it is "start over and hope nothing external was already mutated."

The uncomfortable part is that the defaults almost never get chosen. They get inherited. A container platform ships a logging driver with a size cap, a host ships a journal with a percentage cap, an image ships a logrotate stanza copied from a web server tutorial, and an agent written for those defaults starts producing records that are one to two orders of magnitude larger than a web server ever produced. The cap stays the same while the rate changes, and the recovery horizon silently collapses from weeks to under an hour.

This piece works out what that horizon actually is, why it shrinks as a session grows rather than staying constant, which rotation modes quietly drop the record they were supposed to preserve, and how to size the policy from the task instead of from the disk. The arithmetic is the whole argument, so it is shown rather than asserted.

The Record an Agent Writes Is Not a Log Line

Rotation policies were designed around a specific shape of data: short, self-contained, high-frequency lines. A typical syslog line is a couple of hundred bytes. An nginx access line is roughly the same. A retention cap expressed in tens of megabytes has always been generous against that shape, because tens of megabytes of two-hundred-byte lines is hundreds of thousands of events, and hundreds of thousands of events is a long operational history for a request handler that starts fresh on every request.

An agent does not produce that shape. The unit of work is not a request that starts and ends; it is a session that accumulates. And the thing most frameworks write to the trace is not a status line — it is the model's input and output, which means the full prompt that was sent and the full completion that came back. A single round trip with an eight-thousand-token prompt and a five-hundred-token completion is about thirty-three kilobytes of payload at four bytes per token, before any tool-call arguments, tool results, timestamps or trace identifiers are added. With a couple of kilobytes of tool input and output folded in, one tool call is roughly thirty-five kilobytes of record.

Thirty-five kilobytes per tool call against a two-hundred-byte syslog line is a factor of about one hundred and eighty. That factor is not a rounding error and it is not the user's fault. It is what happens when a logging default designed for small events is pointed at a workload that logs its own reasoning. The cap did not change; the record did.

Rotation Sets a Horizon, Not a Ceiling

The instinct is to read a retention cap as a limit on disk usage, and it is that, but the operational quantity it actually determines is time. Divide the cap by the rate at which the record grows and you get the number of seconds of history the host can still show you. That number is the recovery horizon: the longest gap between "the last thing persisted" and "now" that a restart can bridge without external information.

At thirty-five kilobytes per tool call, the horizon for the common defaults is short in a way that is genuinely surprising the first time it is computed.

The thirty-megabyte row is the one to look at, because it is not a strawman. It is the exact configuration that the most widely deployed container logging driver documents in its own example: a ten-megabyte maximum size with three files retained, which multiplies out to thirty megabytes of total history. That is a perfectly reasonable policy for a web container emitting access lines. Pointed at an agent that calls a model every three seconds, it retains forty-four minutes. An agent that dies at hour nineteen of a twenty-hour task has, on those defaults, no local record of the first eighteen hours. The bytes are gone, and there is no log line anywhere that says they were deleted, because the deletion is the policy working as configured. It is the same shape of failure described in the four sites where a long-running agent accumulates memory, where the constraint that ends a deployment is the one nobody instrumented.

This is the point where the cost asymmetry becomes visible, and it is extreme. Raising that cap from thirty megabytes to two gigabytes consumes five percent of a forty-gigabyte disk. The thirty-megabyte cap it replaces consumed seven hundredths of one percent. Both are rounding error against the disk, and yet one of them decides whether a failed run costs you a resume and the other decides whether it costs you the whole task. Published work on agent-driven checkpoint synthesis measures the second number directly: in a benchmark of sixteen MPI applications, successful checkpoint-and-restart implementations averaged under an hour of agent time and about fifteen million tokens, while attempts that failed on fragmented state consumed more than one hundred million tokens and over three hundred minutes without producing anything usable. The record you delete to save five percent of a disk is the record that decides which of those two outcomes you pay for.

The Horizon Shrinks as the Session Grows

Everything above assumes a constant rate, which is the generous assumption. Real agent sessions do not log at a constant rate, because the context grows. Turn one sends a small prompt. Turn fifty sends the original prompt plus forty-nine turns of history plus tool results plus retrieved documents. Turn five hundred sends all of that again, larger.

If every turn logs the full context it sent, the record does not grow linearly with turns — it grows with the sum of contexts, which is quadratic. Start with a two-thousand-token context and add four hundred tokens per turn, and the cumulative record looks like this.

Averaged across the first hundred turns that is about one hundred and seventy kilobytes per turn, five times the thirty-five-kilobyte figure used earlier. The consequence for rotation is worse than a constant factor, because it changes over time. A policy sized during the first hour of testing, when turns are small and cheap, is sized for a rate the session will not maintain. By the time the session is long enough that losing its history actually hurts, the policy is deleting history several times faster than the test suggested.

There is a second-order effect that matters just as much: the deletion order is oldest-first, and oldest-first is precisely the wrong order for recovery. The turns a crash-recovery procedure needs are the early ones — the original task statement, the constraint that was set at turn three, the list of resources that were already touched. Those are the first records the retention window discards. What survives is the recent history, which is the part a restarted process can most easily reconstruct from the external systems it just touched. The same asymmetry between what is visible and what decided the outcome runs through the slow accumulations that kill a 24/7 agent in week three, where the damage is compounding quietly below every threshold anyone is watching.

The Defaults Were Sized for Someone Else's Workload

It is worth reading the defaults precisely, because two of the most common ones encode an assumption that is wrong for agents in a way that is easy to miss.

systemd's journal, which is where a bare host's application output lands unless someone intervenes, has four relevant directives. The size limits default to a percentage of the filesystem: the "use at most" figure is ten percent and the "keep free" figure is fifteen percent, each capped at four gigabytes, and the journal respects whichever of the two is smaller. The per-file size limit defaults to one eighth of the use-limit, capped at one hundred and twenty-eight megabytes, and the file count defaults to one hundred. Rotation by age defaults to one month per file, and the retention directive — the one that would delete entries older than a specified span — defaults to zero, meaning it is off.

Two things follow. First, on any host with a forty-gigabyte filesystem or larger, both percentage figures hit their four-gigabyte ceiling, so the journal stops scaling with the disk. Second, when the journal does hit its limit it deletes only archived files; active files stay where they are, and the documentation notes that the actual footprint can therefore exceed the configured limit even after a vacuum. Neither behaviour is a defect, but together they mean that "the host has plenty of disk" is not a statement about how much journal history is retained.

The container logging driver is the more direct hazard, and its own documentation is explicit. The maximum file size defaults to negative one, which means unlimited, so the retention count — whose default is one — is only effective once a size is set. An operator who sets only the count and not the size gets a single unlimited file, and one who sets the example values gets thirty megabytes total. The documentation also states plainly that when rolling creates excess files, the oldest file is removed.

That is the whole mechanism in one sentence: the oldest file is removed, and the oldest file is the one holding the task's opening state.

Which Rotation Mode Drops the Record

A retention cap decides how much history exists. The rotation mode decides whether the history that exists is actually the history that was written. These are separate questions, and the second one is where records disappear without any size ever being exceeded.

The classic rename-based rotation is the safe case for a well-behaved writer: the file is moved aside, a new one is created, and the writer is signalled to reopen. It is unsafe for a writer that cannot be told to reopen, because the process keeps its open file descriptor pointing at the inode that was just renamed. Writes continue into that inode — now under the rotated name — while the newly created file sits empty. The journal looks active, the process looks healthy, and every record written after the rotation lands in the archive file that the next rotation cycle is about to delete. The rotation tool's own documentation flags the neighbouring mode for exactly this reason, noting that it exists for programs that cannot be told to close their logfile and might keep appending to the previous one.

The copy-and-truncate mode solves the descriptor problem and introduces a different one. Instead of renaming, the file is copied and then truncated in place, so the descriptor stays valid and the writer keeps writing to the same inode. The cost is a window between the copy and the truncate during which anything written is addressed to a file that is about to be emptied — and the documentation acknowledges the trade-off directly, stating that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. For a web server losing a fraction of a second of access lines is irrelevant. For an agent, that window is where the record of a tool call that already mutated something external would have been written.

There is a subtler failure that has nothing to do with file descriptors. When rotation keeps dated archives and prunes by comparing filenames, the comparison is lexical, and the rotation tool requires that the date format sort correctly as a string — year, then month, then day. A format that puts the day first breaks that ordering, and the consequence is not a cosmetic mislabelling: the pruning step can decide that a newer archive is the older one and delete it while keeping a stale file. The record that mattered is removed and the record that did not is retained, and the retention count looks perfectly correct from outside.

The Signals That Say the Horizon Is Too Short

None of this announces itself. Rotation that deletes the record looks exactly like rotation that preserves it until the day you need the record. Three cheap checks catch most of it before that day.

The first is a coverage gap. If the trace carries a monotonic sequence number or a step index per session, the difference between the highest index recorded and the index the process last reported tells you how many records have already been discarded. A gap that exists a minute after a fresh start is a sign the new session is writing faster than the policy can hold. A gap that appears only after a long run is a sign the rate is growing with the context.

The second is the age of the oldest record for the session you currently care about, as opposed to the age of the oldest file on disk. These differ whenever one session is chattier than another, because a single policy is serving both. Query the store for the specific session identifier and read the timestamp on its earliest surviving record; that number is the horizon for that task, and it is the only number that matters for deciding whether resume is possible.

The third is the rotation event itself, paired with whether the writer reopened. If the trace file was rotated and the process did not reopen a fresh file — visible as a trace path that stops receiving writes while the process continues to work — then records are being written to a descriptor whose name no longer refers to the active file. This is the failure described above, and the distinguishing symptom is that the trace looks frozen and the process does not.

Sizing the Policy From the Task, Not the Disk

The usable move is to invert the calculation. Rather than inheriting a cap and discovering the horizon afterwards, decide how long a task the host must be able to resume, multiply by the rate, and set the cap from the product. The rate is measurable in an afternoon, because it is just bytes written per turn divided by seconds per turn.

The following is the retention required at a modest rate of one tool call every ten seconds, which is the profile of an assistant that answers rather than one that hammers an API.

The reason this table is easy to act on is that the numbers are small. A seven-day recovery horizon at an unhurried rate is two gigabytes — five percent of a forty-gigabyte disk, and the ticket closes there. The reason it is easy to get wrong is that the same two gigabytes looks like an absurd log budget if you are still thinking in web-server units, where two gigabytes is millions of access lines. The unit confusion, not the disk, is what produces the thirty-megabyte default.

One refinement makes the budget mostly unnecessary. The local file only has to cover the crash-recovery window — the time between the last durable state and the failure — while the complete record has to survive for audit and for post-hoc analysis, and those are different requirements with different price points. Splitting them lets the local journal stay small and fast while the full trace is shipped off the host to object storage, where capacity is cheap and retention is a lifecycle rule rather than a rotation policy. A local horizon of a few hours and an archived record of ninety days satisfies both jobs for less than either job would cost on its own.

Why This Belongs on a Host You Control

The reason the horizon is a decision rather than a constraint is that someone has to own the file. On a managed function platform the trace goes to the platform's logging service, the retention window is a console setting with a hard ceiling, and the file descriptor question does not arise because there is no file. That is genuinely convenient, and for short-lived handlers it is the right trade. It stops being the right trade the moment the workload is a session that runs for hours and whose recovery depends on records written at its beginning, because at that point the retention window is a product decision that someone else made for an average workload, and the average workload is not an agent.

What makes the host the right unit here is that the policy is executable on it. The rotation mode is a configuration choice, the retention cap is a number you set, the reopen behaviour is a signal your process either honours or does not, and the shipped archive is a lifecycle rule you write. All four are yours, which means the recovery horizon becomes something you specify rather than something you discover after an incident. The workloads this matters for are the long-lived ones: an always-on agent holding a session across hours, a trading monitor that must be able to prove what it saw and when, a document or data pipeline whose batches are long enough that restarting from zero wastes real money. Those are the shapes where a fixed retention ceiling set for someone else's average becomes a liability.

This is the configuration a dedicated cloud server exists to provide, and where a prebuilt deployment saves the setup: an instance where you own the filesystem, so the journal limits, the container driver options and the rotation stanzas are all yours to set, with a preconfigured image that gets the runtime online in about three minutes instead of an afternoon. OpenClaw on SurferCloud is offered in that shape — a month-to-month lightweight instance at 6.75 USD, a first-month UHost promotional tier at 6.90 USD, and a first-quarter tier at 4.66 USD, each with 40GB of SSD, an independent IPv4 address and a choice of Japan, Singapore or United States data centres. Instances range from 1 vCPU with 2GB up to 4 vCPU with 8GB, which is the right order of magnitude for a single always-on agent plus its trace store, and the disk is the one you configure. The specific numbers change with promotions, so treat them as the current page state rather than a permanent price.

Two practical notes about sizing for this use case, since the trace is what sets the disk floor. The lightweight tier's 30 Mbps peak bandwidth and the UHost tiers' dedicated bandwidth are both far above what trace shipping needs — the archive upload is measured in megabytes per day, not per second — so bandwidth is not the constraint and should not drive the choice. Storage is. An agent plus a few gigabytes of local journal plus a week of retention fits comfortably in the smallest tier; the upgrade becomes worthwhile when the archive grows past what a lifecycle rule can trim, or when several agents share one host and their horizons add up.

FAQ

Is this the same problem as the disk filling up?

No, and conflating them is why the fix gets misapplied. A full disk is a capacity failure and the response is to add capacity or shrink the writers. A short recovery horizon is a durability failure, and adding capacity does not fix it if the retention count is what limits the history. A host with a terabyte free and a three-file rotation policy still retains forty-four minutes. The mechanics of the capacity side are covered in the four distinct failures behind a full disk, and only one of them is a retention decision.

Do I need the whole prompt and completion in the trace?

For recovery, no — you need the decisions, the tool calls, their arguments and their outcomes. For debugging a wrong answer you need the prompt. If the two requirements are in conflict on a small host, keep the compact decision record locally with a long horizon and ship the full payload archive elsewhere with a shorter local presence. What you must not do is keep only the chatty version locally and let it consume the horizon.

Does compression solve the size problem?

It reduces it, and it is worth enabling, but a rotation tool's compressed archives can only be pruned after they exist, and the pruning step is the same one that deletes the oldest file. Compression changes the constant, not the order of deletion. It also interacts with the descriptor problem: if compression is deferred to the next cycle because the writer might still be appending, the most recent history is deliberately uncompressed and therefore larger than the steady-state calculation suggests.

What is the single fastest thing to change?

Compute your own rate — bytes per turn divided by seconds per turn — and set the size cap from the horizon you want instead of accepting the default. That is a one-line change on most hosts and it is the change that converts recovery from a hope into a property. Everything else on this list is an improvement; that one is the difference between having a record and not having one.

Is the archive a backup?

No. A shipped trace archive is a record of what happened, not a copy of the state that resulted. It can tell you which operations were attempted and which completed; it cannot restore the database those operations touched. The distinction is the same one behind why a snapshot is not a backup — a copy that shares the failure domain of the original is not a recovery path. Treat the trace as the input to a recovery decision, not as the recovery itself.

The Point of the Policy

A rotation policy is the only part of agent durability that costs almost nothing and is almost never specified on purpose. It is invisible while it works, silent when it fails, and its failure mode is not an outage but an inability to explain one. The move is small: measure the rate, invert the calculation, decide how many hours of history the host must be able to show, and set the cap and the mode from that number rather than from a tutorial written for a web server. An agent that can resume is not the one with the most sophisticated checkpoint table; it is the one whose record of the first hour survived until the nineteenth.