HTTP 502 Bad Gateway: The Five Backend States Behind It

A 502 Bad Gateway almost never means your application code is broken, and in the four architectures that produce most of them — nginx in front of an app, an ALB in front of a fleet, a CDN in front of an origin, a Kubernetes ingress in front of pods — the failing component is one of five distinct backend states: a TCP listener that is gone, a listener that is present but forbidden, a listener that is up but returning garbage, a listener that is dying between requests, or a listener that was never the address you configured at all.

That redefinition is what makes the error tractable. A 502 is a statement about a connection, not about a service. The proxy accepted your request, dialed the upstream, and the conversation failed at the TCP, HTTP, or policy layer — before a status code from the application could reach the response. RFC 9110 defines it narrowly: the gateway received an invalid response from the inbound server while acting as a proxy. "Invalid" covers dead, refused, malformed, mid-stream-truncated, and protocol-mismatched, which is exactly why a single "is the app up?" check never resolves it.

What follows is a field triage sequence that identifies which of the five backend states you are in within about five minutes, using one log line and two socket commands. Then each state gets its own section: the log signature that proves it, the command that confirms it, the fix, and the failure mode that makes people misdiagnose it as one of the other four. A closing comparison table separates 502 from its neighbours 504, 503, 500 and 104, because half of all wrong fixes start as wrong status-code reads.

State 1 — the listener is gone: "connection refused" behind your 502

The direct answer: when nginx logs connect() failed (111: Connection refused) while connecting to upstream, the backend process is not accepting connections on that port at all, and the fastest proof is curl -v http://127.0.0.1:8080/health from the proxy host — a refusal in milliseconds, not a timeout.

This is the most common 502 in the wild and the most commonly mislabeled one, because the error surfaces at the proxy while the cause sits one hop behind it. As of October 2026 the failure pattern documented across production postmortems is consistent: a deploy, a crash, an OOM kill, or a config reload that changed the listen port, and the application container or process simply is not there anymore. The TCP stack answers on behalf of a dead process — with RST — which is why the failure is instant rather than slow.

The distinguishing evidence is the error string and the timing together:

• (111: Connection refused) — nothing is bound to the address:port pair, or the bind is on a different interface than the one nginx dials (127.0.0.1 vs the pod IP).
• Failure arrives in under 10 ms — the kernel rejected it; a network or firewall problem would consume the connect timeout instead.
• systemctl status or docker ps on the backend host shows the unit exited, restarting, or absent.

Confirm the state, in order:

1. From the proxy host, run curl -v --connect-timeout 3 http://<upstream>/ and watch for "Connection refused" — this proves the proxy-to-backend path is the broken hop.
2. On the backend host, run ss -tlnp | grep <port> — an empty result means no listener; a listener on 127.0.0.1 while the proxy dials the routable IP means a bind-address mismatch.
3. Check journalctl -u <service> --since "10 min ago" for a crash or OOM: an OOM-killed process leaves "Killed process ... oom-kill" lines that the application's own log will never show.
4. Restart or roll back, then re-run step 1 before touching any nginx configuration — the config was almost certainly fine.

The classic misdiagnosis here is treating a repeated restart crash-loop as a proxy problem, reloading nginx over and over while the application dies faster than the eye can follow. Read the journal first; a crash-looping unit has a visible restart count.

State 2 — present but forbidden: security groups, firewalls and the silent drop

The direct answer: if the upstream host is alive and listening but a stateful filter between proxy and backend drops the SYN, the proxy sees a connect timeout, and what it emits is a 502 (or a 504, depending on which timer expires first) — the state to check is the network policy, not the application.

Security groups and cloud firewalls are the second-largest source of 502s in autoscaling fleets, and the reason is structural: security groups are applied at instance launch, so a scale-out event can introduce a fresh instance whose group differs from the template the old instances were built from. The signature is load-dependent and intermittent — some requests succeed, then a burst of 502s, then recovery, correlating with which backend the load balancer chose.

How to separate a filter drop from a dead process with two commands:

1. curl -v --connect-timeout 3 telnet://<ip>:<port> from the proxy host — an instant refusal is State 1; a hang until the 3-second timer expires is a dropped packet, which is State 2.
2. sudo iptables -L -n -v | head -30 (or the provider's security-group console) on the backend host — look for a REJECT/DROP rule covering the proxy's source IP, or a group that only allows a stale CIDR range.

Three sub-causes account for most of the real incidents: the security group allows the office CIDR but not the proxy's private subnet; the group allows port 22 but not the application port; or an NACL — which is stateless — permits outbound return traffic to be dropped even though inbound was allowed. The NACL case is the nastiest because single-packet probes (SYN) pass while full connections die, producing 502s that only appear under real traffic.

Fix by widening the source filter to the proxy's address or subnet, then re-run the connect probe. Do not fix by disabling the firewall, even temporarily on a production host — the correct permanent state is a rule that names the proxy.

State 3 — up, but speaking the wrong protocol: TLS, HTTP/2 and header mismatches

The direct answer: when the error log shows SSL_do_handshake() failed, upstream sent invalid response, or recv() failed (104: Connection reset by peer) on a connect that succeeded, the backend answered in a dialect the proxy cannot parse — a TLS mismatch, a plaintext answer to an HTTPS dial, or an HTTP response with headers the proxy rejects.

This state is characterised by a successful TCP handshake followed by immediate failure, and it clusters around three concrete mismatches. A proxy configured with proxy_pass https:// against a backend serving plaintext gets a TLS alert or garbage bytes in reply. A backend whose certificate is expired or whose chain is incomplete fails verification and gets reset. And an HTTP/2 upstream that resets streams under concurrency — RST_STREAM with code 0x8 — surfaces as intermittent 502s on exactly the requests that arrived on reused connections, as documented in nginx upstream triage guides through October 2026.

The probe that disambiguates is protocol-aware, so use the same dialect the proxy uses:

1. openssl s_client -connect <upstream>:443 -servername <sni> — verify the chain validates, the protocol negotiates, and the certificate's notAfter date is in the future.
2. curl -v https://<upstream>/health (or http:// if that is what nginx dials) — confirm the response starts with HTTP/1.1 200 or similar, and that no header exceeds the proxy's limits.
3. Compare with the nginx directive: if proxy_pass says https:// and the backend speaks HTTP, or the reverse, one word in the configuration is the entire incident.
4. For HTTP/2-to-HTTP/2 resets, capture and decode: tcpdump -i any -w /tmp/up.pcap 'tcp port 443' then tshark -r /tmp/up.pcap -Y 'http2.type == 3' -T fields -e http2.error_code — code 0x8 (REFUSED_STREAM) means concurrency limits; forcing the proxy leg to HTTP/1.1 with proxy_http_version 1.1; is the documented quick isolation step.

One header variant deserves its own warning: a backend response with a header larger than the proxy's buffer settings, or an invalid character in a header name, produces "upstream sent too big header" or "upstream sent invalid header" — this is State 3 even though the application is perfectly healthy. The fix is either correcting the backend's emission or sizing proxy_buffer_size/proxy_buffers to match reality, never disabling buffering wholesale to mask it.

State 4 — dying between requests: timeouts, keepalive races and upstream resets

The direct answer: when 502s are intermittent, arrive under load, and the error log shows upstream prematurely closed connection or recv() failed (104: Connection reset by peer) while reading response header, the backend is closing idle keepalive connections at the same moment the proxy reuses them — the connection is alive when nginx picks it and dead by the time the request lands on it.

This is the race condition class of 502, and it exists because two independent timers govern one connection: the proxy's keepalive idle timeout and the backend's. If the backend's timeout is shorter — the default keepalive timeout for many application servers is in the 5-to-60-second range while nginx holds idle upstream connections for 60 seconds by default — then any connection idle past the backend's limit is closed server-side while remaining in nginx's pool. The next request routed to it gets a reset mid-conversation.

The canonical fix is to make the proxy expire first, with margin:

1. In the upstream block, enable connection reuse: keepalive 32; (or sized to expected concurrency, e.g. 64–128 for a busy proxy).
2. Set the proxy leg to HTTP/1.1 with cleared Connection headers: proxy_http_version 1.1; proxy_set_header Connection ""; — without this, keepalive to the upstream never engages.
3. Size the idle timeout: in an upstream with keepalive_requests and zone settings, or simply keep the backend's idle timeout at least 1.5–2× the proxy's, so the proxy always retires the connection before the backend does.
4. For slow endpoints, raise proxy_read_timeout above the backend's true worst-case processing time — but measure first: a blanket 300s timeout trades 502s for pile-ups, and the request-capacity math is the same one behind long-running workloads that exhaust a host in week three.

The evidence that you are in State 4 rather than State 1 is temporal, not textual: upstream_response_time in the access log shows small values (the reset came fast, the connection was warm), failures cluster a few seconds after traffic dips, and direct curl against the backend always succeeds because each curl opens a fresh connection and never races the idle pool.

State 5 — wrong address: DNS, stale configs and the phantom upstream

The direct answer: when the 502 persists while the backend is provably healthy, the proxy is not talking to the backend at all — it is dialing a resolved address that is wrong, stale, or no longer routed, and the proof is comparing the DNS answer on the proxy host against the address in the error log.

DNS failure behind 502s has a property that catches almost everyone: it appears after an infrastructure change and survives the fix that inspired it. The mechanism is caching at multiple layers. nginx resolves upstream hostnames once, at configuration load, by default. If the target's address changes — a migration, a re-created load balancer, a pod reschedule — the running proxy keeps dialing the old address until it is reloaded, and if the old address is now assigned to something else, the responses that come back are wrong in ways that read as application bugs.

Three sub-patterns, each with its own give-away:

• Static resolution, changed target. The fix is nginx -s reload or a variable in proxy_pass with a resolver directive, which forces runtime resolution — the standard pattern for Docker Compose and Kubernetes sidecars, documented in upstream routing guides current as of October 2026.
• TTL-blind clients in autoscaling. A service name resolving to an endpoint list that rotated; the proxy holds A records past their TTL. Symptom: 502s that resolve themselves after exactly the DNS TTL elapses.
• Container IP reuse. The old pod's IP is reassigned to a different service; the proxy connects and gets a valid but wrong response. Symptom: 502s mixed with 404s or wrong-content responses — the wrong-content responses are the tell that separates State 5 from every other state.

Diagnose in one minute:

1. dig +short <upstream> on the proxy host — the address the proxy should be using now.
2. Compare with the upstream_addr field in the error log entry — a mismatch is the whole finding.
3. Reload the proxy or add the resolver directive; re-test.

The disk-space cousin of this state deserves a mention because it mimics a network problem: a proxy host whose disk has filled cannot write its own error log, and helper processes fail in ways that surface as 502s. When nothing in the five states checks out, run df -h on the proxy before anything else — a full disk is four distinct failures, and a wedged proxy is one of them.

The five-minute triage sequence that identifies your state

The direct answer: read one error log line, run one connect probe, run one listener check — in that order — and the result maps one-to-one onto a state; the sequence is deliberately ordered so each step eliminates a state instead of merely gathering information.

Run it in this order and stop at the first conclusive result:

The ordering principle matters more than the commands: State 1 is both the most common and the cheapest to confirm, so it goes first; DNS (State 5) is rare but cheap to check last; the middle states need the connect probe because their log signatures overlap.

Fixing 502 vs 504 vs 503: read the status code before you touch anything

The direct answer: a 502 means the conversation with the upstream failed before a usable response existed; a 504 means the upstream was reachable but too slow; a 503 means the proxy or an explicit health decision refused to route the request at all; and a 500 means a real response from the application that says "I broke" — conflating these is the single most expensive diagnostic error in this domain.

Before any comparison, the practical conclusion: if the status you see is 504, the five states above are the wrong list — you want timeout and capacity analysis; if it is 503, look at health checks and capacity signals; only a genuine 502 puts you in connection-failure territory. Now the full mapping:

The CDN wrinkle: when a content delivery network such as Cloudflare sits in front, a 502 page branded by the CDN means the CDN's own fetch failed, and headers like cf-error-type distinguish edge-generated errors from passed-through origin errors. A Cloudflare-branded 502 with a 522 in the underlying status is a connect timeout to your origin — State 2 territory — while a plain 502 passed through unbranded means your own proxy emitted it and the five states apply unchanged.

The alert worth setting before the next 502

The direct answer: alert on the proxy error log, not on the 5xx rate — a 5xx alert tells you users are already seeing failures, while an error-log alert on connect() failed and upstream prematurely closed fires while the failure is still a probe, before the retry budget is exhausted.

The concrete instrumentation is small. Log the upstream interaction with an explicit format — nginx supports an upstream_log format recording $upstream_addr, $upstream_status and $upstream_response_time — and alert on two conditions: any occurrence of (111: Connection refused) in the error log (State 1, zero tolerance), and a sustained share of upstream_response_time above the read timeout (State 4 developing). Both are single-line log checks any monitoring stack can evaluate, and as of October 2026 this log-format-first approach is the standard recommendation in production triage guides.

The deeper prevention is architectural, and it is the same conclusion the five states push you toward: single-backend deployments turn every backend state into an immediate client-facing 502, while two or more healthy backends behind one proxy convert most incidents into retries that no user ever sees. That is the strongest practical argument for running the proxy tier against a pool — which is also the configuration in which State 4's keepalive race and State 5's DNS staleness must be tuned deliberately rather than discovered incidentally.

Where the architecture matters: single box vs cloud fleet

The direct answer: on a single self-managed box, all five 502 states are local and the triage sequence above is the complete fix; on a cloud fleet, States 2 and 5 dominate because filters and addresses change underneath a running proxy, and the fix belongs in the deployment path — security groups in templates, DNS names with short TTLs, health checks wired to remove backends automatically.

The two contexts differ in which state bites you, not in what the states are:

For teams running the single-box shape — one lightweight instance fronting its own application, the configuration most small deployments and side projects actually use — the practical difference is that the triage sequence runs entirely on one host and takes minutes. That shape also makes rehearsal cheap: a 502 lab (kill the backend, tighten the group, shorten the backend keepalive, swap the DNS record) exercises four of the five states deliberately, on hardware whose loss costs nothing, and the configuration that survives the lab is the configuration you copy to production.

When a lightweight instance is the right place to rehearse this

The direct answer: the fastest way to internalise the five states is to reproduce all of them on a disposable machine where a 502 costs nothing, and a lightweight cloud instance — SurferCloud's ULightHost line is built for exactly this scale of work — provides that environment at the price point that makes leaving it up for a week of experimentation rational.

SurferCloud positions ULightHost for small deployments and individual developers, and the relevant properties for a 502 lab are the ordinary ones: root access to install nginx and an application server, a public address to reproduce State 2 by filtering your own traffic, and a console to observe the backend while the proxy fails. The annual promotion pricing — 50% off as of October 2026 per the current promotion page, with entry configurations from roughly USD 1 per month — is the part that turns "rehearse failure modes" from a scheduled exercise into something you can leave running. The current offer details and configuration tiers are on the ULightHost promotions page.

What the lab buys you is not the commands — those are above — but the recognition speed. The difference between a fifteen-minute outage and a two-hour one is rarely tooling; it is whether the person reading the error log has seen (111: Connection refused) in a context where they could experiment, versus meeting it for the first time at 2 a.m. with production traffic failing. Five states, one disposable server, one evening each, and the next 502 is a lookup instead of an investigation.

Frequently asked questions

"My site returns 502 — is my server down?" Not necessarily, and the distinction is the entire point of the five states. The proxy is up by definition (it answered you). The backend is in one of five specific states, four of which leave the application itself healthy. Run the five-minute sequence before concluding anything is down.

"I restarted nginx and the 502 went away. Why?" Because the reload re-resolved the upstream hostname (State 5) or reopened connections that were racing the backend's keepalive (State 4). It will return unless you make the fix structural: runtime DNS resolution via a resolver directive, or a timeout hierarchy where the proxy expires connections first.

"Why do I get 502 only under load, never when I test manually?" That is the signature of State 4 — your manual curl opens a fresh connection and never touches the idle pool that is racing the backend's keepalive timer. Fix the timeout ordering, do not raise limits blindly.

"Cloudflare shows my site as 502 — is it their problem?" Check whether the error page is branded and what the underlying status is: a Cloudflare-branded 502 with a 522 underneath means their edge could not reach your origin — your State 2 or a routing problem on your side. An unbranded 502 is your own proxy's answer, passed through.

"Should I just increase all the timeouts?" No. Timeouts fix 504-shaped problems and the keepalive race specifically. A connection refused (State 1) or a security-group drop (State 2) is unaffected by any timer, and blanket 300-second read timeouts convert failures into pile-ups. Measure the actual response-time distribution first.

"How do I tell 502 from 504 quickly during an incident?" One curl with response headers. 504 means the proxy waited and gave up — capacity or slowness. 502 means the conversation failed structurally — the five states in this article. The proxy error log names which one in its first line.

"Is there a config that prevents all five states?" No configuration prevents a backend crash (State 1). What prevents all five states from becoming client-visible incidents is a pool of healthy backends behind the proxy, health checks that remove failing ones, and the alert on the error log described above — detection plus redundancy, not a magic directive.

The short version

A 502 is a statement about a connection, not a verdict on your application. Five backend states produce nearly all of them: a gone listener that refuses instantly, a filtered path that times out on connect, a protocol mismatch that resets after handshake, a keepalive race that dies between requests, and a wrong address that was resolved once and never again. One error-log line, one connect probe and one listener check identify which state you are in, in that order, in about five minutes. The fixes are small and specific; the prevention is a backend pool plus an error-log alert. And the cheapest time to learn all five is on a machine you can break on purpose — a lightweight instance left up for a week of deliberate failure teaches more than a year of reading postmortems.