No Space Left on Device Is Four Failures, Not One

A full disk is not one failure. It is four different failures that print almost the same error, and the command most people run first can only see one of them.

The message is familiar from every layer of a stack. A database refuses a write. A container runtime reports it while starting a container. An application logs it and exits. Deployments stall. It is the same five words, and it names the symptom rather than the cause: there is no space left on a device, on the filesystem that device is formatted with, for the operation that was attempted. Teams that read it as "the disk is full" reach for one of two reflexes, delete something or buy a bigger volume, and neither of those fixes three of the four cases. One of them routinely makes the next outage worse.

This is a diagnostic piece. It sets out one measurement worth taking first, then a table of failure classes that all produce the same message and are routinely confused with each other, then what to change so the problem stops recurring.

The Command Everyone Runs Measures Less Than They Think

df -h reads the filesystem's own accounting. It reports how many data blocks are allocated, how many are free, and how much of that free space is available to a normal user. That is a narrow answer to a narrow question. It does not tell you how many inodes are still free, whether a file that was deleted an hour ago is still being held open by a process, or whether the space that does remain is writable by the account your application runs as. Those are three separate questions.

The percentage in the Use% column is the single most over-trusted number in server operations, because it answers "are the data blocks allocated" and nothing else. A filesystem at 60 percent can be completely unable to accept a new file. A filesystem at 100 percent can still accept writes from the right process. Neither of those statements is a rounding error; both are normal behaviour, and both are documented.

The One Signal Worth Taking First

Before changing anything, take the gap between two numbers: what the filesystem says it is using, and what the directory tree can actually account for.

df -h gives the first number. du -xhd1 / | sort -h gives the second, walking the tree, staying on one filesystem, and listing directories one level deep so you can descend into whatever is largest. When the two agree within a few percent, the space really is held by files you can see and reach. When df reports substantially more used than du can find, the space is held by something with no name in the directory tree at all.

Then run df -i and read the IUse% column, not the block percentages. This is the check almost nobody automates, and it is the one that catches the failure class that survives on a volume with plenty of free space. A filesystem has a fixed number of inodes, each one describing a file or directory, and exhausting them stops file creation just as completely as exhausting data blocks does.

Those two commands, df -i and the gap between df and du, separate the four cases below. Everything else is confirmation.

Four Failures That Produce the Same Message

Failure 1: Ordinary Growth Is the Only One a Bigger Disk Fixes

The unglamorous case, and the one the reflexes are built for. Logs, uploads, database files and container images accumulate until the volume is genuinely full. df and du agree, df -i looks fine, and the largest directory from the du walk is exactly what you expect it to be.

This is the one class where adding capacity is a real answer, provided you also understand the growth rate. A volume that filled over nine months needs a capacity decision. A volume that filled in four hours needs a retention decision, because whatever wrote that data will fill the larger volume too, only later and more expensively.

Failure 2: inode Exhaustion Hides Behind a Healthy Percentage

Filesystems create inodes when they are made, not on demand. An ext4 volume formatted with defaults has one inode for roughly every sixteen kilobytes of capacity, which is generous for photographs and tight for session files. A workload that creates a very large number of very small files, zero-byte session records, cache shards, mail queue entries, temporary uploads, spends inodes far faster than it spends space.

The resulting incident is distinctive and unpleasant. A published postmortem from August 2026 describes exactly this shape: dashboards reporting roughly 40 percent free space on an ext4-backed fleet while database and file writes failed across production, traced to millions of zero-byte session files consuming filesystem metadata rather than capacity. Standard capacity alerting passes the whole way through, because it only watches the number that is not the problem.

The same class appears at container scale. Kubernetes exposes inode pressure as an explicit eviction signal, nodefs.inodesFree, with a default hard threshold of 5 percent, which is a direct acknowledgement that a node with free blocks can still be out of file handles.

Reasonable people get this wrong because the symptom points at the wrong resource. The service reports no space; the dashboard reports free space; the argument starts. The resolution is to add df -i to whatever you already look at, because the two percentages move independently and only one of them was on your screen.

Failure 3: The Space Is Already Deleted

A file has two things that can outlive each other: a name in a directory, and an open file descriptor in a process. Deleting the name does not necessarily free the data, and the system call documentation is explicit about it. When the last link to a file is removed and no process has it open, the file is deleted and its space becomes available for reuse. When the last link is removed but a process still holds the file open, the file remains in existence until the last file descriptor referring to it is closed.

That is the mechanism behind the most confusing version of this incident. Someone notices a large log file, deletes it, and df does not move. The directory entry is gone, so the file no longer appears in du, but the writer, which opened the file at startup and never reopened it, is still appending to an inode that no longer has a name. The space is unreachable through the filesystem and unrecoverable through deletion.

The diagnostic is lsof -nP +L1, which filters for open files with a link count below one, and shows the process, the descriptor, and a name ending in (deleted). Two cautions when reading it. Several processes and descriptors can hold the same file, so do not sum the sizes in the output directly. And the reported size is not the allocated block count, so sparse files will mislead you.

The clean fix is to make the process reopen its output. For a service with a documented reopen mechanism, that is a reload of the writer rather than a restart: Nginx, for example, reopens its log files on a benign signal, and its log rotation documentation describes this as the supported way to release the old handle. For anything without that mechanism, the choice is a service restart at a controlled moment, or a fix to the rotation configuration so the writer reopens by itself next time.

There is a faster and more dangerous option: truncating the file through the process's own descriptor under /proc. It releases the space immediately, and it will also happily truncate something that is not a log file. If a descriptor points at business data, you have replaced a full disk with data loss. This is a last resort under a deadline, not a routine move.

Failure 4: The Five Percent You Forgot

ext family filesystems reserve a slice of their capacity for the super-user. The manual page for the tool that creates them is direct about the purpose: the reserve exists so that root-owned daemons can keep functioning correctly after non-privileged processes have been prevented from writing to the filesystem. The default is 5 percent.

Two consequences follow, and both catch people out. First, the reserve is counted as used, so a freshly formatted ext4 volume commonly reports a few percent used before you have written anything at all, and every capacity calculation carries that permanent deduction. Second, and more importantly, the reserve is not available to your application. A service running as an unprivileged account hits the wall at 95 percent of the volume, while root can still write. If your monitoring samples as root, or if you tested the fix by hand as root, everything looks fine and the service still fails.

When this is the cause, the honest question is not how to reclaim the 5 percent. It is why the service is running as an account that needs that margin removed. Clicking the reserve down with a filesystem tuning tool is a legitimate emergency lever, and a poor permanent arrangement on a volume that is expected to run near capacity, because the reserve is what stands between a full filesystem and an unusable one.

Where the Space Actually Goes: Two Loggers

In practice, most of these incidents are manufactured by one of two loggers, and both ship with defaults that permit unbounded growth.

The first is the system journal. On a persistent systemd journal the size ceiling is governed by configuration that defaults to 10 percent of the filesystem, capped at 4 GB, with a separate default of 15 percent of the filesystem to be left free for other uses, and the journal respects whichever of the two is smaller. The important detail is what happens when a filesystem fills up around it: the journal stops using more space, but it does not go back and remove existing files to reduce its footprint, and only archived journal files are eligible for deletion. A journal that grew while you were not looking does not shrink on its own once the pressure eases.

The second is container logging. In the default file-based logging driver, the maximum size of a log before it is rolled defaults to unlimited, and the maximum number of log files defaults to one. A container that writes to standard output without a rotation policy therefore writes without a ceiling. The fix is a pair of logging options in the daemon configuration, and two details matter: the values must be provided as strings, and the change applies only to newly created containers, so an existing container keeps the old behaviour until it is recreated.

Both of these belong in your configuration and not in your incident response. A ceiling set in advance is the difference between a log that grows to a limit and a log that fills a volume at three in the morning.

In Kubernetes the Failure Changes Name

The kubelet watches node resources and terminates pods to reclaim them before the node starves. The default hard eviction thresholds are worth knowing exactly, because they define the operating envelope: available memory below 100 MiB on Linux nodes, node root filesystem available below 10 percent, image filesystem available below 15 percent, and free inodes below 5 percent on either. Thresholds are evaluated on a short housekeeping interval, and a hard threshold means a zero-second grace period, so termination is immediate rather than drained.

Two properties of this mechanism are routinely misread. Eviction is a protection measure, not a cleanup. The kubelet reclaims what it can, removing unused images before it touches workloads, but the data that filled the filesystem is still sitting there. A replacement pod scheduled onto the same node arrives at the same full disk, and the cycle repeats; teams often describe this as a flapping node when it is a full one. And per-pod ephemeral storage limits are evaluated separately from node pressure, so an eviction message that names a container's local storage limit is a different problem from a node reporting disk pressure, with a different fix.

The Permanent Fix Is Rarely Capacity

Three of the four failure classes are indifferent to how much storage you have. A larger volume gives inode exhaustion more room before it bites and does not otherwise change the mechanism. A larger volume does not close the file descriptor that is holding deleted data. A larger volume does not stop an unrotated log from growing; it only lengthens the interval between outages and makes the eventual bill larger. Buying the bigger shared instance to absorb a log burst is paying for the symptom rather than removing it, and the same burst returns on the new plan.

What actually removes the failure classes is unglamorous and cheap:

• A rotation policy with a retention limit on every writer, whether that is the system journal, a container logging driver, or an application log directory.
• A retention policy for the file types your workload creates in bulk, so session files, shards and queue entries have a defined lifetime rather than an accidental one.
• inode monitoring alongside capacity monitoring, alerting on IUse% and not only on Use%.
• An open-but-deleted check in your routine, because the condition persists silently for as long as the process runs.
• Deliberate headroom, so that the difference between a warning and an outage is large enough to be an actionable window.

None of those require a larger machine. All of them require that you are able to make the decision at all.

When You Own the Box

Every lever in this article so far, the size of the super-user reserve, the inode count the filesystem was created with, how long logs live, whether you can make a writer reopen a file descriptor without a full restart, is a decision that exists only when you control the machine. On a fully managed application platform most of those decisions are made for you, which is a real convenience right up to the moment you exhaust the storage the platform allows. At that point the failure is still yours, but the instruments are not: you can observe the symptoms and you cannot reach the causes. The four-way diagnosis above collapses back into a support ticket.

That is the specific case a plain virtual server is built for. It is not the only case, and it is not always the right answer, but when a workload is long-running and its failure modes are storage retention and file handles rather than request volume, a normal Linux or Windows box where you own the filesystem, set the retention, and can act on all four classes is the configuration that actually closes the problem.

The ULightHost promotional page at this link is the lightweight end of that line: plans from $2.9 per month with up to 62 percent off, in over 16 locations, with network and storage figures of up to 10M PPS and 1.2M IOPS quoted on the page. For an operations workload, the terms carry more weight than the headline. The page states that these plans do not support changing region or resizing, that deleting a resource mid-term is refunded at the original list price, that each plan is limited to one server per customer, that addresses are datacenter IPs with no replacement option, and that port 25 is blocked by default. It also distinguishes the two pricing modes on the page: the Essential tier at $2.9 per month is a new-user price that renews at the regular rate, while the remaining monthly tiers are listed with the renewal price locked, with the page naming the Lite plan as the exception. Those caveats are not small print to work around; they are the same properties that decide whether a plan suits a service you intend to leave running.

If a single volume with a rotation policy and an inode alert would close your last three incidents, that is the shape of machine worth renting, and the caveats above are the ones to check against your workload before you do.

FAQ

Why does df show free space when writes fail with ENOSPC?

Because df -h reports data blocks only. Free blocks coexist with exhausted inodes, with space held by deleted-but-open files, and with capacity reserved for the super-user. Run df -i and compare df against du before concluding anything about capacity.

Does deleting a log file free the space immediately?

Only if no process still holds it open. If the writer opened the file at startup and is still appending, the data survives with no directory entry until the last descriptor closes, which usually means a reopen or a restart of that process.

Is a bigger disk the fix?

For ordinary data growth, yes. For inode exhaustion, held-open deleted files, and unrotated logs, no. Those three will consume the larger volume on the same schedule, or reproduce identically at the same usage level.

How do I monitor for inode exhaustion?

Track IUse% from df -i per filesystem, not just the block percentage, and alert on both. If you run Kubernetes, the default inode eviction threshold is 5 percent free on the node and image filesystems, which is a useful reference for where a comfortable margin ends.

What is the fastest safe fix when a production volume is already full?

Identify the class first, because the fix differs. For deleted-but-open files, make the writer reopen its output. For a runaway log, cap it in the writer's configuration and rotate. For inode exhaustion, remove the small-file population at its source. Truncating a file through a process descriptor releases space instantly and is legitimate only when you have confirmed the descriptor points at a log and not at data you need.

Sources

• Kubernetes documentation, node-pressure eviction reference, including default hard thresholds for nodefs.available (10 percent), imagefs.available (15 percent) and inode availability (5 percent) and the zero-second grace period for hard thresholds.
• mke2fs manual page, documenting the -m reserved-blocks-percentage option and its 5 percent default, and the intended purpose of the super-user reserve.
• unlink manual page, documenting the behaviour of a deleted name that a process still holds open.
• systemd journald.conf manual page, documenting the default journal size ceiling of 10 percent of the filesystem capped at 4 GB, the 15 percent keep-free default, and the behaviour when a filesystem fills around the journal.
• Docker documentation for the json-file logging driver, documenting the default unlimited max-size, the default max-file of 1, and the string-value requirement in the daemon configuration.
• State of Cloud Reliability 2026, an analysis of 47 major public cloud incidents from 2017 to 2025, reporting that configuration and change error accounts for 45 percent of them and that median incident duration is 3.4 hours.
• HAMS Tech, incident postmortem on Linux inode exhaustion, published 17 August 2026, describing production write failures on ext4-backed instances while capacity dashboards reported roughly 40 percent free.
• OneUptime, diagnosing DiskPressure and inode evictions caused by pod ephemeral storage, published 27 August 2026, on separating pod storage-limit eviction from node pressure.
• Tencent Cloud developer community, a Kubernetes pod eviction troubleshooting handbook published 9 October 2026, on the eviction sources and the log-rotation cause behind disk pressure.
• A df/du/lsof troubleshooting write-up published 10 October 2026, on distinguishing block exhaustion, inode exhaustion and deleted-but-open files in order.

See the current ULightHost plan details and terms