iowait Is Not a Disk Metric: One Counter Divided by Your Core Count

A high iowait percentage does not mean your disk is slow, and a low one does not mean it is fast. The counter measures how many tasks were waiting on I/O while a CPU had nothing else to run — which means it moves with your core count at least as much as with your storage, and on a wide enough machine it can sit permanently near zero while every request on the box is dying.

This is not a subtle caveat buried in a footnote. The Linux manual page for /proc/stat — the file every monitoring agent reads to produce that percentage — states plainly that the value "is not reliable" and then lists three separate reasons why. Most dashboards graph it anyway, as a single line labelled "disk". That single line is the most common reason teams replace perfectly healthy storage, and, in the opposite direction, the reason a genuine storage bottleneck survives weeks of investigation.

What follows is what the counter actually counts, why it is capped by your vCPU count, why it can decrease when conditions get worse, which counters do describe the device, and a five-minute sequence that identifies a real storage bottleneck without guessing.

What iowait actually counts

Every entry in /proc/stat is a duration measured in USER_HZ units — hundredths of a second on nearly every architecture. The kernel's own field list is short and each line is one sentence long. iowait, introduced in Linux 2.5.41, is defined as "time waiting for I/O to complete."

Read that next to the definition of idle: "time spent in the idle task." The two fields are both about CPU time. The percentage your dashboard shows is iowait divided by the sum of all the cpu line fields. It is a share of processor time, not a measurement of device service time. No part of that arithmetic touches the disk's queue depth, its completion latency, or its queue count.

The practical consequence appears the moment a task blocks. When a process issues a read and goes to sleep waiting for it, the CPU that was running it becomes idle. Two things can happen next, and the kernel's choice between them is what makes the metric unreliable:

1. If no other task is runnable, the CPU idles. The time is billed to iowait.
2. If another task is runnable, that task is scheduled on the CPU immediately. The time is billed to whatever that task does — user, system, or another iowait when it blocks.

The manual page spells out the second case as the first of its three reasons the value is unreliable: the CPU will not wait for I/O to complete, so when it goes into idle state for an outstanding task's I/O, another task gets scheduled on it. In other words, iowait accumulates only when the machine is genuinely out of other work. It measures the intersection of two conditions — blocked tasks and idle cores — and reports it as though it measured one.

The counter is divided by your core count

Because the percentage is a share of total CPU time, the same storage behaviour produces a different number on every instance size. One task fully blocked on I/O consumes all of one core's time, so it can drive iowait to at most 1 / N of the total, where N is the number of logical CPUs.

This is the dilution effect, and it is why the metric is worse the larger your instance gets. Suppose you run a service with 400 concurrent workers and each worker spends 3% of its wall-clock time blocked on a disk read. That is 12 task-equivalents blocked at any moment. On an 8-vCPU instance the counter saturates — it reads 100%, because there are more blocked tasks than cores and every core has something waiting. On a 64-vCPU instance the identical workload reads 18.75%. Same storage, same latency, same 400 workers. The number fell by more than five times because you resized the instance.

That is not a hypothetical distortion at the edges. It is the normal condition of any horizontally scaled service on a modern wide instance. Resizing upwards is a common first response to a performance complaint, and one of its side effects is to make the storage signal associated with that complaint disappear from the dashboard.

There is a second, subtler form of dilution. Two hundred tasks each blocked 0.5% of the time produce the same average of one task-equivalent as a single task blocked 100% of the time. Both read 12.5% on an 8-core instance. The user-visible symptom, however, is completely different: one machine has a single request stuck for seconds, the other has a small constant delay applied to every request in flight. The counter cannot tell you which machine you are looking at.

The value can go down when the disk gets slower

The third bullet on the manual page is the one that should end the practice of alerting on this metric: the value in the field may decrease in certain conditions. It is not an error or a wrap-around. It falls out of the accounting model.

The kernel attributes CPU time to a single state per tick. If a task blocks on I/O and the core is otherwise idle, the tick is credited to iowait. If a background cron job, a log shipper, or a garbage collection thread wakes up and becomes runnable during that same period, the tick is credited to user or system instead. Nothing about the disk changed. Load on the machine changed, and the storage evidence was overwritten by unrelated CPU work.

This produces the scenario that costs teams the most time: a storage layer degrades, blocked tasks accumulate, that accumulation wakes up monitoring and retry threads, those threads consume the freed CPU time, and iowait falls. The dashboard shows an improvement. The latency graph from the application tells a different story, but it is on a different screen, owned by a different team.

Because the field is computed per CPU as well as system-wide, the multi-core case adds a third distortion. The manual page notes that a task waiting for I/O is not running on any CPU, so its contribution to each individual core's iowait is difficult to calculate. Per-core iowait, in a container that sees a slice of a host, is not something to reason about at all.

The three counters that do describe the device

The kernel exposes real device statistics in /proc/diskstats and in /sys/block/<device>/stat. Both files contain the same seventeen fields; /proc/diskstats prefixes each row with the major and minor device numbers and the device name. All fields except one are cumulative monotonic counters — you compare two snapshots and divide by the elapsed time.

Field 11 is the one worth understanding. The documentation describes it as incremented at each I/O start, completion, merge, or read, by the number of I/Os in progress multiplied by the milliseconds spent doing I/O since the last update. The result is a quantity that rises proportionally to both how long requests take and how many are stacked up behind them. A device with fast service and a deep queue and a device with slow service and a shallow queue can produce the same weighted time, which is exactly why it is described as a measure of completion time and backlog together rather than either one alone.

Reading the cumulative counters means taking two snapshots. The kernel field list warns that applications reading these counters should detect and handle resets, because the remaining fields reset at boot, on device reattachment or reinitialisation, or on counter overflow. A monitoring job that stores only the derived rate will silently produce a negative or absurd value across a reboot unless it stores the raw counters too.

%util reaches 100% on the first request, not the hundredth

Field 10 is what iostat turns into %util, and the iostat manual page defines that column as "the percentage of elapsed time during which I/O requests were issued to the device." Combined with the kernel's wording — the field increases so long as field 9 is nonzero, which means so long as at least one request is in flight — the effect is a saturation indicator with a floor of one.

A single 4 KB read in flight sets the counter running. If that read lasts a millisecond out of every ten, %util reads 10%. Nothing about the queue depth, the device's parallelism, or its spare capacity enters the calculation. And on the devices most cloud instances now ship with, the manual page adds the sentence that matters: device saturation occurs when the value is close to 100% "for devices serving requests serially," but for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.

So on a modern NVMe device, %util at 100% means one of the device's many concurrent slots is occupied. It does not mean the device is out of capacity. The inverse reading is also true and more dangerous: a device that is genuinely saturated — its internal queues full, requests waiting in the block layer — can show a modest %util if the host-side view only ever sees one request at a time from a single-threaded application. Concurrency you never generate is concurrency the counter never observes.

One multiplication turns two counters into a queue length

Because the raw counters are in the same file, you can derive the number you actually wanted without any inference from iowait. It is Little's law applied to the block layer: the average number of requests in the system equals the arrival rate multiplied by the average time each spends in the system.

Track that product over time and the two failure modes separate cleanly. Rising latency at flat throughput is a device getting slower. Flat latency at rising throughput is headroom being consumed, which is healthy right up until the queue depth stops being absorbed. Falling throughput with rising latency is the beginning of a collapse, because the block layer is spending its time managing a queue rather than moving data.

The reason to compute the product rather than read aqu-sz alone is that the derived figure can be attributed. If you know which application threads are issuing the I/O, you can compare their arrival rate against the device's. If the numbers do not reconcile, the extra requests are coming from somewhere you have not accounted for — a backup job, a snapshot, a log shipper, or an encryption layer issuing more requests than the caller believes it is sending.

await is two numbers added together

The iostat manual page defines await as the average time for I/O requests issued to the device to be served, and then adds the clause that makes it hard to read: this includes the time spent by the requests in queue and the time spent servicing them. It is a total, not a service time.

That distinction decides which fix is correct. Queueing delay is produced by your own concurrency and disappears if you submit less work at once. Service time is produced by the device and does not. Both appear in the same column.

The shape in that table is the useful part. Service time does not move. Everything above it is queueing, and queueing is a function of how much concurrency you generated. So when await doubles, the first question is not whether the storage got slower — it is whether the arrival rate changed. A deploy that adds 40% more database connections reproduces that table without the storage layer changing at all.

The honest limitation of the table is that it is a model. Real devices serve requests in parallel, so the wall does not arrive at 100% utilisation; it arrives when the device's own internal queue saturates. That is the reason to measure rather than assume, and the reason to push load against the storage before you need to know the answer.

The signal that measures what iowait was supposed to measure

Pressure Stall Information exists because the kernel developers reached the same conclusion. The PSI documentation states the problem directly: when CPU, memory, or IO devices are contended, workloads experience latency spikes and throughput losses, and without an accurate measure of that contention, users are forced either to play it safe and under-utilise their hardware, or to roll the dice and frequently suffer the disruption.

PSI exports two lines per resource in /proc/pressure/ — cpu, memory, and io — each with avg10, avg60, avg300 rolling averages and a cumulative total in microseconds.

Three properties of this interface solve the problems in the previous sections. First, it is not divided by core count: io.pressure full reads the share of time the workload was stalled, so one stalled task on a 64-core machine can read 100% where iowait reads 1.56%. Second, it cannot be masked by unrelated CPU work, because the denominator is non-idle tasks rather than all CPU time. Third, the absolute total counter in microseconds lets you detect latency spikes that never make a dent in the time averages, which the documentation calls out as an explicit design goal.

Because PSI is tracked per cgroup, it answers the question a container platform actually faces: not whether the host's storage is under pressure, but whether this workload is being starved by it. A noisy neighbour saturating the device shows up in your cgroup's io.pressure while the host-wide iowait line stays flat and uninformative.

A five-minute sequence, in order

Run these in sequence and stop at the first one that resolves. Each step costs seconds and rules out a class of explanation.

1. Read /proc/pressure/io and compare some against full. If full is materially above zero, the workload is genuinely stalling on storage and you can stop suspecting the metric and start investigating the device. If full is flat at zero but some is elevated, a subset of tasks is blocked while the machine is otherwise productive — the same condition iowait was averaging into invisibility.
2. Take two /proc/diskstats snapshots ten seconds apart. Compute field 3 and field 7 deltas for bytes, fields 1 and 5 for completed operations, fields 4 and 8 for summed service time. Divide to get average service time per direction. Read and write latency are frequently an order of magnitude apart on the same device, and a single blended await hides which one is the problem.
3. Compute the queue depth product. Multiply operations per second by average latency as in the earlier table, and compare against the device's own concurrency. A queue depth of 30 on a device that serves 1 concurrently is a saturated device; the same depth on a device that serves 128 concurrently is idle.
4. Check fields 16 and 17, flush requests and milliseconds spent flushing. A database writing with durability enabled spends much of its latency budget here. Flush stalls are serialised by the block layer, so they behave differently from ordinary writes and cannot be diagnosed from read/write counters.
5. Attribute the arrival rate. If the derived queue depth is higher than the applications you know about can produce, the surplus is coming from another source on the box. Backup jobs, snapshot creation, log rotation, and container image pulls all issue I/O that no application metric accounts for.
6. Repeat before and after any change. Because the counters are cumulative and reset at boot, the only way to compare two configurations is to measure both from a known boot state or to store the raw counters alongside the derived rates.

Five things that look like a slow disk and are not

The reason a storage investigation takes days rather than hours is rarely the measurement. It is the presence of at least four other faults that produce the same symptom and are easier to blame on hardware. Each of these has a distinguishing signal.

The first row is the most expensive. When service time is flat and observed latency is climbing, the storage layer has not changed and there is nothing to buy. Adding capacity reduces the queue depth by spreading requests across more devices, which lowers observed latency — and it looks like a fix, because the number improves. The concurrency that produced the queue is still there, now spread wider, and it will re-saturate the larger device at a higher arrival rate.

The third row is the one most often mis-attributed in the other direction. A shared-vCPU instance suffering from steal time produces slow requests that never touch the disk slowly. The device counters will be unremarkable. Iowait may read high, because blocked tasks plus idle CPU is exactly what the counter is designed to report — and the blocked tasks are waiting on a deliverable CPU, not on storage at all.

Distinguishing the device from the layer above it

The four minutes of measurement above tell you whether the device is the constraint. The harder question is what to do when it is, because "the disk is slow" is rarely one condition. A device can be limited by its own service time, by the concurrency the host generates, by the block size the workload uses, or by the fact that reads and writes have very different characteristics on the same hardware.

The order that costs the least is worth stating explicitly, because the instinct is to skip to the last item. First, separate reads from writes using fields 4 and 8 — a workload that is write-latency-bound needs a different fix than one that is read-latency-bound, and the blended column hides which you have. Second, separate durable from non-durable operations using the flush counters — a database configured for full durability is measuring the flush path, and that path is serialised by design. Third, check whether your concurrency exceeds what the device serves in parallel, which is a queueing problem and not a hardware problem. Fourth, check the request size distribution, because a device rated in tens of thousands of operations per second is rated that way at a specific block size, and small random requests are a different workload. Only after those four does provisioning enter the picture.

What changes when the answer really is infrastructure is the floor under your latency, and that floor is easy to look up rather than estimate. The trend worth knowing about is that storage has stopped behaving like the rest of computing. In its fourth-quarter 2026 memory outlook published on 21 October 2026, TrendForce projected storage contract prices rising a further 15% to 20% quarter over quarter, with enterprise SSD contract prices up 23% to 28% — the only major category in the industry where the rate of increase was accelerating rather than easing. Meanwhile the same reasoning that has made storage the expensive axis has made serving it cheaper: Cloudflare reported on 23 October 2026 that rebuilding its range-request cache cut p99 time-to-first-byte for its object storage roughly in half and reduced inter-datacentre traffic for those workloads by about 6%.

Read together, those two data points describe the trade you are actually making. The bytes are getting more expensive while the software that moves them is getting better, which means the traditional answer of "throw more devices at it" now carries a price signal it did not carry two years ago. That is an argument for measuring accurately before provisioning, not an argument against provisioning.

This is the point at which the choice of platform starts to matter more than the choice of instance size. A provider that publishes the storage characteristics of its fleet lets you check the assumption that the previous sections told you to measure. Lightweight cloud instances in the SurferCloud ULightHost line are documented on the promotional page at up to 10M PPS for network performance and 1.2M IOPS for storage read and write performance, with NVMe disks on every plan from the entry tier upward. Those two numbers are the right shape of thing to compare — operations per second at a stated block size, alongside a network packet rate — rather than a single vague claim about speed.

They are also worth reading in the context of everything above. A published IOPS figure is a ceiling for the device under ideal request sizes, and it is the piece of information most providers do not give you. If you have spent the last twenty minutes establishing that your problem is queue depth at 4 KB request sizes, knowing the device's parallel-request capacity is the number that tells you whether more concurrency will help.

Choosing a small instance without repeating the measurement

There is one practical complication worth naming. The measurement sequence above is cheap on a live production machine and awkward during a purchasing decision, because you cannot run /proc/diskstats snapshots against hardware you have not bought.

Two things close that gap. The first is that the diagnostic works identically on a small instance — nothing in the sequence depends on core count, which is the whole point of moving off iowait. You can provision the cheapest configuration that resembles your target, run the six steps, and get numbers that transfer.

The second is the pricing structure you run it inside. On the ULightHost promotional plans, the entry configuration is listed at $2.9/month on a new-user deal with a renewal at $7.5/month, and the widely used mid-tier configurations — 2 vCPU with 4 GB of memory, or 4 vCPU with 4 GB — are listed at $10.4/month and $19.5/month respectively, with the page stating that renewal price is locked for all users except the Lite plan. Every plan carries a dedicated IPv4 address and an NVMe disk. That structure matters for this particular measurement because the evaluation is short and bounded: you need a machine for a few hours to generate a request-size distribution and read the resulting counters, not a long-term commitment.

The page's stated terms are worth reading before you plan around them: the activity does not support changing zones or configurations, and if resources are deleted midway the refund is calculated on the original price. Each plan is limited to one server per customer, the plans use datacenter IPs with no replacement option, and port 25 is blocked by default. Those constraints are consistent with a workload you size once and hold, which is also a reasonable description of the trial-and-measure workflow described above.

Frequently asked questions

Should I alert on iowait at all? Not as a primary signal. The value is explicitly documented as unreliable, it is divided by your core count, and it can fall while conditions worsen. If a legacy dashboard requires a CPU-derived storage line, alert on io.pressure full instead and keep iowait as a historical reference. The kernel's own PSI documentation frames the metric as the answer to under-utilisation versus overcommitment, which is a better framing than "disk load."

Why does iowait read high on an idle machine with a slow disk and low on a busy one? Because it measures the intersection of blocked tasks and idle CPUs. On an idle machine there are no other runnable tasks, so every blocked microsecond is billed to iowait. On a busy machine a runnable task is scheduled immediately and that same storage delay is billed to user or system. The counter is telling you about scheduling, not about storage.

Is %util ever useful? On a single rotating device or a serialised queue it is a reasonable saturation indicator, because the device serves one request at a time and the counter's floor matches the device's limit. On SSDs and RAID arrays, the manual page is explicit that the number does not reflect the device's performance limits. Use it as a yes/no indicator of whether any request was in flight, and get the depth from the weighted field or from the throughput-times-latency product.

Does PSI replace the disk counters? No, and the two answer different questions. PSI tells you whether your workload is being starved and by how much — the impact. /proc/diskstats tells you what the device is doing — the mechanism. Diagnosis needs both: PSI to confirm the storage layer is the constraint rather than a symptom, and the counters to decide which of the four fixes applies.

What is the fastest way to find out whether a slowdown is storage or CPU? Compare service time from fields 4 and 8 against observed latency. If summed service time divided by completed operations is stable while await grows, the extra time is queueing and the device is not slower. If service time itself has moved, the device changed. Those are two different investigations with two different budgets, and the two counters that separate them sit in the same file.

The number to stop graphing

Iowait has survived in dashboards because it is convenient: it is one line, it comes free with CPU metrics, and it is named after the thing people suspect. Every property that makes it convenient makes it misleading. It is a share of CPU time rather than a device measurement, it is divided by the core count of whatever instance it runs on, it is overwritten by unrelated CPU work, and the kernel's own documentation tells you it is unreliable and can decrease.

The replacement is not another single number. It is two files: /proc/pressure/io to establish that your workload is being starved, and /proc/diskstats to establish what the device is doing about it. Read together, they separate a device that got slower from a queue that got deeper, a read problem from a write problem, a durability stall from a bandwidth limit, and a storage bottleneck from a CPU one. The whole sequence takes about five minutes and replaces a metric that has been quietly wrong for years.

If you are sizing a small instance and want to run the measurement before committing, the ULightHost promotional plans start at $2.9/month with NVMe storage and a dedicated IPv4 on every tier, and the plan comparison is on the ULightHost promotional page. If you would rather compare against enterprise-grade configurations with dedicated bandwidth, the Hong Kong node cloud server page lists the full specification set.

For the CPU side of the same problem, reading the three load average numbers correctly explains why that metric also counts tasks rather than time. When the storage failure is one of exhaustion rather than latency, no space left on device is four failures, not one covers the three causes that leave free bytes on the disk. And when a shared instance is the suspect rather than the disk, CPU steal time shows how identical specifications produce different performance, while a full connection tracking table covers the network-layer failure that drops packets while the link looks idle.