A Full Connection Tracking Table Drops Packets While Your Link Is Idle

A full connection tracking table will stop a server that has bandwidth to spare, which is why exhaustion so rarely reads as a sizing mistake and so often reads as an attack.

That is the whole problem. Every other capacity limit on a Linux host announces itself in a graph someone is already watching. Disk fills up and the usage line bends toward the ceiling. CPU saturates and the load average climbs. Bandwidth saturates and the interface counters flatten at line rate. Connection tracking has no such line on the default dashboard. It fills silently against a number almost nobody has looked up, on a path that carries a few hundred kilobits per second at the moment the first drop happens, and the only symptom the operator sees is that some unknown fraction of new connections never establish. There is nothing to correlate, so the first instinct — that this is traffic-shaped — is also the wrong one.

This is a walk through what the table is, what actually fills it, the single counter that tells you it is full rather than merely busy, and the four other failures that present almost identically. It ends with a sizing worksheet and a fix list ordered by cost.

Your link is bored while your table is full

The useful way to picture connection tracking is as a fixed-size hash table that the kernel consults before it decides whether a packet belongs to a conversation it has already approved. Every packet entering the netfilter stack is looked up against that table. If the entry exists, the packet is part of an established flow and the firewall's stateful rules are cheap to evaluate. If the entry does not exist, the kernel decides whether this is a new flow worth tracking, and if the answer is yes, it allocates an entry. That allocation is the moment of scarcity.

When the table is full, the kernel has no place to put the entry. It does not panic, and it does not log an error to the application. It drops the packet and increments a counter. Depending on which kernel path you hit, that shows up as an nf_conntrack: table full, dropping packet line in the kernel ring buffer, or as a rising insert_failed value that never reaches any dashboard by default. Meanwhile the link utilisation graph stays flat and unremarkable, because the packets being dropped are small — a SYN is 60 bytes with a typical option set — and they are being dropped before they ever became meaningful volume.

Put a number on it. An attacker or an unlucky client population that can push 50,000 new connections per second will, against a five-minute establishment timeout, need 15 million entries to keep them all alive. Nobody's table is that large. But even at a fraction of that rate the arithmetic is unforgiving, and it does not require anyone to be attacking you at all. A fleet of mobile clients that reconnect aggressively on every network change produces the same shape of load without the intent.

This is the reason the failure is so often mislabelled. The symptom arrives as a partial outage, the fingerprint resembles a flood, and the volume that caused it can be measured in single-digit megabits. The link was never the constraint.

What actually consumes the table

Five sources fill connection tracking in normal production, and only the fifth is adversarial. Most exhausted tables in the field are a combination of the first four.

Long-lived idle connections that never close. The default established timeout is 432,000 seconds — five days — per the kernel's own sysctl documentation. That figure sounds absurd until you consider what it is protecting: a TCP connection can be silent for hours and still be perfectly alive, and the kernel has no way to distinguish that from a connection whose peer vanished without sending a FIN. So it holds the entry. Every pooled database connection, every idle WebSocket, every SSH session in a detached terminal, every long-poll request a load balancer is holding open occupies a slot for up to five days after the last packet moved.

Outbound UDP, which has no handshake to end it. DNS lookups default to a 30-second timeout, and the extended stream timeout for recognised UDP conversations is 120 seconds. These look small next to five days, but UDP is where the multiplier bites: every distinct five-tuple — source address, source port, destination address, destination port, protocol — is a separate entry, and there is no connection setup or teardown to regulate how fast they accumulate. A service that makes one DNS query per request, with a fresh ephemeral source port each time, generates one table entry per request and holds it for thirty seconds after the answer arrives. QUIC, which moves HTTP traffic to UDP, multiplies this further, because every browser session is now a stream of UDP flows rather than one TCP connection.

Container networking, which duplicates every conversation. On a host running containers, a single packet can be tracked more than once as it traverses the host's bridges, iptables rules and any overlay encapsulation. ClusterIP service routing performs destination NAT at the node, and every such translation is stateful. A host that looks quiet when you count application connections can be carrying several times that number of tracking entries purely as an artifact of how the traffic is forwarded.

Outbound NAT from a shared address. This is the pattern that exhausts tables fastest at scale, because the failure mode is inverse. A gateway or egress host representing thousands of internal clients must create one entry per outbound connection, and those entries live as long as the connection does. Where a single server's table scales with its own concurrency, an egress host's table scales with the concurrency of everything behind it — and the ceiling does not move.

Scanning traffic and deliberate state exhaustion. The adversarial case is the one that gets named in incident reports, and it is the least interesting technically. A scanner walking your port range creates an entry per probe. A spoofed-source flood creates entries for conversations that will never become real. Two details make this cheaper for the attacker than it looks: entries for half-open connections ages out on a much shorter timer — nf_conntrack_tcp_timeout_syn_sent defaults to 120 seconds and syn_recv to 60 — so a spoofed flood has to be sustained rather than bursty; and a single attacker can trigger entries for many distinct five-tuples using one packet each, so the cost ratio is heavily in their favour.

Note what the last point implies. State exhaustion needs very little bandwidth, but it does need the packets to arrive, which means it is a form of availability attack that a bandwidth-based mitigation tier will not intercept. That distinction matters when you buy protection, and it matters more when you diagnose one.

One entry is two entries

There is a detail in the kernel documentation that invalidates most back-of-envelope sizing, and it is easy to miss because it sits at the end of a paragraph about something else. Entries are added to the table twice — once for the original direction and once for the reply direction, with the reversed address — which means that with default settings a maxed-out table has an average hash chain length of two, not one.

Read that as an operator and the consequence is immediate: the number of distinct connections a table can hold is half the number of entries it can hold. If your nf_conntrack_max reads 262,144, you do not have room for 262,144 connections. You have room for roughly 131,072, because every approved conversation consumes both directions.

The sizing rule the same documentation gives for the hash table is also worth internalising, because it explains why two machines with the same memory behave identically and badly. The bucket count is calculated by dividing total memory by 16,384, clamped to a floor of 1,024 buckets and a ceiling of 262,144. Work that ceiling backwards and the interesting result falls out: a 4 GB machine computes 262,144 buckets and a 16 GB machine computes 1,048,576 — which gets clamped to exactly the same 262,144. Above roughly 4 GB of RAM, the default hash table size stops growing. Buying a bigger instance does not buy a bigger table unless something explicitly changes the sysctl.

Here is the same arithmetic as a table, with the caveat that distributions override these defaults and you must read your own values rather than assume:

The practical lesson: on a 2 GB instance and a 16 GB instance with untouched defaults, the connection ceiling can differ by a factor of four while the memory differs by a factor of eight. Instance sizing and table capacity are not the same purchase.

The one signal worth watching

There is a counter that separates "the table is busy" from "the table is full," and it is not the one people look at first. Comparing nf_conntrack_count against nf_conntrack_max tells you how close you are, and a healthy-looking ratio is genuinely reassuring — but a table sitting at 60 percent with a rising insert failure count is already dropping traffic, and the ratio alone will never show it.

The signal is the insert failure counter, read from conntrack -S:

conntrack -S
cpu=0  found=0 invalid=130 insert=48211 insert_failed=1187 drop=1187
       early_drop=0 error=0 search_restart=0

Read it as four numbers that matter. insert is how many entries were successfully allocated. insert_failed is how many allocations the kernel refused because the table had no room. drop is the packets lost as a consequence. early_drop is a specific and telling case: the table was not yet full, but the kernel evicted an existing entry to make room, which means some conversation that was already approved just lost its state. A non-zero and climbing insert_failed is the only unambiguous confirmation you will get, and it is the number to put on the dashboard.

Three commands give you the full picture in under a minute, and they belong in the runbook verbatim:

• sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_buckets — current usage, ceiling, and hash table width in one line.
• conntrack -S — insert and failure counters, per CPU.
• conntrack -L | head -50 and a count by state — what is actually in there, and whether the balance looks like long-lived established flows or a pile of short-lived UDP entries.

The distribution of states is the part people skip and the part that identifies the cause. A table dominated by ESTABLISHED entries with long ages points at pooled connections that are not being recycled. A table dominated by UNREPLIED UDP points at outbound name resolution or a scan. A table dominated by short-lived entries in TIME_WAIT points at a client that is churning connections rather than reusing them. Those are three different fixes, and the counter alone does not distinguish them.

Four failures that look exactly like this one

This is where most investigations go wrong, because connection tracking exhaustion has an unusually convincing set of lookalikes. All four of the following produce intermittent, hard-to-reproduce connection failures on a host whose headline metrics look fine.

There is a fifth lookalike worth naming separately because it is not a lookalike at all: a genuine state-exhaustion attack. It produces exactly the same counters as a self-inflicted capacity problem, and the difference is only in the source distribution. Before you escalate to a mitigation provider, look at the top talkers in the table. If a handful of addresses dominate, it is deliberate. If the sources are spread across hundreds of ordinary-looking clients, you are being failed by your own sizing, and no amount of upstream scrubbing will change the ceiling.

The same principle governs how to interpret latency symptoms in shared-resource environments: two instances can report identical specifications and perform differently for reasons that are entirely outside the guest's counters, which is what makes the CPU steal time measurement worth adding to the same dashboard. Knowing which layer is lying to you is most of the diagnostic work.

Fixes ordered by what they cost

There is a meaningful distinction between raising the ceiling and removing the load, and the order below reflects it. Start with the free changes.

Shorten the established timeout — this is free and it is usually the largest single win. Moving nf_conntrack_tcp_timeout_established from its five-day default to a few hours discards the entries that are almost certainly not protecting anything real. The risk is a connection that is genuinely idle for longer than your new value being dropped mid-conversation, so the number has to be chosen against your longest legitimate idle period rather than picked at random. For a workload whose connections are pooled and actively reused, a value in the low hours is defensible. For one holding long-lived sockets open across long silences, it is not.

Raise the ceiling, and raise the hash table with it. Increasing nf_conntrack_max alone makes the table longer without making lookups faster; the bucket count is what keeps the chains short, and the conventional guidance is to keep buckets at a fraction of the maximum rather than a one-to-one ratio. Every entry costs memory, so the increase is a real purchase and not a free setting — which is exactly why raising it without first finding out what is filling the table is a bad trade. You will pay the memory cost and hit the new ceiling a month later.

Stop tracking traffic that does not need tracking. For flows that will never be filtered statefully — a bulk replication feed, a backup stream, a known-good high-volume service on a dedicated port — a NOTRACK rule in the raw table before netfilter's stateful hooks removes the entries entirely. This is the structurally correct fix where it applies, and it is also the one with a sharp edge: NOTRACK makes the connection invisible to -m state matching, which breaks any rule, security group or NAT behaviour that depends on it. Apply it in a test environment first, and never to traffic whose return path depends on tracked state.

Reduce the multiplier rather than the ceiling. If the table is full of short-lived UDP entries from name resolution, a local caching resolver on the host collapses thousands of upstream queries into far fewer long-lived ones. If it is full of churning client connections, connection pooling and HTTP keep-alive on the egress path reduce the number of distinct five-tuples in flight. Both are application or configuration changes rather than kernel tuning, and both are more robust than any ceiling you can raise.

Move away from iptables-based service routing in clusters. Where container networking is the consumer, replacing per-packet iptables DNAT with an eBPF-based data path removes a large class of connections from tracking entirely, because the translation no longer needs a stateful table. This is the largest change on the list and the one with the clearest structural payoff in a dense cluster.

One note on what not to do: do not disable connection tracking wholesale to solve this. Unloading the module takes stateful firewall matching, NAT and most security-group implementations down with it. A host that has stopped dropping packets because it has stopped inspecting them is not fixed.

A sizing worksheet you can run today

Do this before you are under pressure, because every step is easier while the table is healthy.

1. Read the three values: nf_conntrack_count, nf_conntrack_max, nf_conntrack_buckets. Halve the maximum to get the real connection ceiling, and compare that to the count.
2. Break the current table down by state and by age. The distribution tells you whether the consumer is long-lived, churning, or short-lived.
3. Identify the top source and destination addresses. A concentrated source suggests scanning; a broad distribution suggests your own baseline.
4. Size the legitimate peak. Take the highest concurrent connection count from the last ninety days, add growth headroom, and check it against the halved ceiling with margin to spare.
5. Set the timeout against your longest real idle period, not against a default you inherited.
6. Instrument insert_failed. If nothing alerts on this counter, the next exhaustion is invisible until a customer reports it.

Steps one through three take a few minutes and answer the question that determines everything else: is this table being filled by traffic you can stop tracking, or by traffic you must serve?

Where the ceiling stops being yours to raise

Everything above assumes the table is yours to size, and for a physical or dedicated host that is true. The picture changes when the connection state is held at a layer you do not administer — a provider's edge, a managed load balancer, a shared NAT gateway, or a hypervisor's virtual switch. In that arrangement you can raise nf_conntrack_max inside the guest until the memory runs out and still lose connections, because the entry that failed to allocate was never in your guest. The counter that proves it is on someone else's equipment.

That is the case a Hong Kong cloud node is built for, and it is why the question worth asking of any provider is not about peak bandwidth. What is the connection-tracking ceiling at the edge, is it per-tenant or shared, and is the insert failure count observable to me? A node adjacent to mainland China and the Southeast Asian mobile markets carries exactly the client population that reconnects aggressively — mobile clients changing networks, cross-border sessions that traverse several networks, and Web3 workloads whose clients are frequently behind carrier-grade NAT with unpredictable source ports. The tracking load that produces is not a bandwidth problem and never will be, and it arrives on a link that shows no strain while it happens. The same reasoning applies to keeping recent platform changes from requiring a reboot: the mechanism matters more than the label when you are deciding whether a fix can be applied without interrupting tracked flows — see why live patching is not the same thing as a reboot replacement for where that distinction bites.

SurferCloud's Hong Kong node page addresses this layer explicitly. The offering is positioned for exactly the workload profile above — Web3 DApp hosting, blockchain node deployment, wallet services, NFT platforms and smart contract APIs, alongside AIGC, software development and cross-border e-commerce — and states that cloud hosts are created within minutes with flexible configuration that can be scaled as needs change, which matters when the fix for a state ceiling turns out to be more headroom rather than better tuning. The protection list is stated at the host and platform level: host intrusion detection, website vulnerability detection, DDoS attack protection and data encryption, on Tier 3 data centres across 17 or more availability zones spanning four continents, with VPC network segments and subnets for logical isolation and a stated 99.95 percent availability commitment.

Two things on the page are worth reading carefully in the context of this article, because they describe capacity that is maintained rather than merely promised. It states that the platform supports kernel hot patch technology, so that during a security incident the platform can be upgraded online without restarting cloud hosts — relevant because a tracked connection in this class of system dies when the guest restarts, and an upgrade path that avoids the restart avoids flushing the table. And it states that the same page's plan cards currently read as sold out, which is a stock state rather than a permanent one; plan availability on a node like this rotates, so check the page or your account manager for current stock rather than assuming this week's status.

The commercial terms on the page at the time of writing are an offer of up to 25 percent off with no price increase on renewal, and payments by bank card, PayPal or cryptocurrency including BTC and USDT. A free trial is offered through an account manager for first-time users. The page does not publish per-plan pricing for the Hong Kong node, so treat any figure you see elsewhere as unverified: the plan cards render specifications, not a public price list, and a full inventory view of pricing and regions is served from the console rather than from the page.

Whether this is the right shape for your workload depends on one number from the worksheet above. If your legitimate concurrent connection count with growth headroom fits the provider's per-tenant ceiling with room to spare, the tracking layer stops being the thing that wakes you up. If it does not, no amount of tuning inside the guest will help.

Start with the three commands. The answer usually changes what you were planning to buy.

Frequently asked questions

How do I know the table is full rather than just busy? Look at insert_failed and drop from conntrack -S, not at the utilisation ratio. A table at 60 percent with a climbing insert failure count is already losing traffic, and the count-versus-max ratio cannot show it. Non-zero early_drop means the kernel evicted an approved entry to make room, which is a state loss even though nothing reported an error.

Can a full connection tracking table affect traffic that is already flowing? Yes, in two ways. New connections cannot be established at all, which is the obvious half. Less obviously, once the kernel starts evicting existing entries, traffic that was already approved loses its state and subsequent packets in that flow are treated as new or invalid, which can drop an established connection that the application believes is healthy.

Is this always an attack? No, and the assumption costs time. Long-lived pooled connections held against a five-day default, UDP from name resolution, and container forwarding can fill a table with no attacker involved. Check the source distribution before escalating: a concentrated handful of sources suggests deliberate exhaustion, a broad spread of ordinary clients suggests your baseline was never sized.

Why does raising the instance size not fix it? Because the default bucket count is derived from memory divided by 16,384 and clamped at 262,144, which means the default hash table stops growing somewhere around 4 GB of RAM. Above that point a larger instance has the same default tracking capacity as a smaller one until something changes the sysctl explicitly. You are buying compute, not connection state.

Should I just disable connection tracking? No. It is load-bearing for stateful firewall matching, NAT and most security-group implementations, and removing it trades dropped packets for uninspected ones. The defensible reductions are targeted — a NOTRACK rule for specific high-volume flows that never need stateful treatment, a local caching resolver to collapse UDP entries, connection pooling to reduce churn — not unloading the module.

What belongs on the dashboard? Three series and one alert. Plot nf_conntrack_count, the halved maximum as your effective ceiling, and the ratio between them. Alert on insert_failed increasing at all. The first three tell you how much runway is left; the fourth is the only one that tells you that you have already run out.