HTTP 500 After Deploy: Rollback-First Troubleshooting Order

A 500 error that appears within minutes of a deploy is a suspect with fingerprints on it, and the correct response is to restore the last known-good version before you understand the bug — because rolling back takes seconds to minutes, debugging takes minutes to hours, and customers experience the difference in full.

That redefinition is what the argument depends on. "Rollback-first" is not the claim that the deploy is always the cause, nor that root-cause analysis does not matter — roughly 80 percent of incidents follow a deploy, a configuration change, or a traffic shift, and a 500-spike that starts within the deploy window is overwhelmingly likely to be the deploy. The claim is about ordering: restoring service and explaining the service are different jobs, and the one that customers experience should finish first. Teams that debug a live outage instead of reverting it routinely turn a 10-minute regression into a 90-minute incident, because "the fix is just one line" is the sentence that precedes most prolonged outages on record. The trade-off that makes this non-trivial is the irreversible migration: a rollback that crosses a forward-only schema change can corrupt data, so the rollback decision needs a 5-minute structure, not a reflex.

What follows is the operational playbook: the exact order of the first five minutes, the four signals that confirm the deploy as the cause, the decision matrix that separates full rollback from partial revert and forward-fix, the migration-reversibility test that decides which of the three you can safely use, the release-engineering patterns — expand-and-contract migrations, feature flags as kill switches, blue-green switchover — that make rollback a 90-second operation instead of a 30-minute scramble, and the anti-patterns that convert a small regression into a second, worse incident. A comparison table separates rollback from partial revert and from forward-fix, because picking the wrong one of the three is how teams destabilize a system that was already healing.

What a 500 Actually Tells You After a Deploy

The direct answer: a 500 means the application itself raised an unhandled exception, and a 500 that starts within the deploy window is a defect in the code you just shipped — a dependency version conflict, a null path the new code did not handle, a config value the new code reads differently, or a schema mismatch — none of which are fixed by restarting the process.

The status code is a coarse signal, but it is not useless. A 502 or 504 means the process is down or unreachable — crash loops, OOM kills, upstream timeouts — while a 500 means the process is alive and its code is failing on real traffic. That distinction matters because it changes the first fix: a crash-looped process may be recoverable by reverting the deployment object, while an unhandled exception in live code must be reverted at whatever layer the new code was introduced. The four defect classes that dominate post-deploy 500s:

1. Dependency or environment drift — the build pulled a different transitive dependency version than the one tested, or the target runtime lacks a system library the new code requires, producing import-time or first-call exceptions.
2. Unhandled edge in new logic — the new code path assumed a shape of input, credential, or feature state that some fraction of production traffic violates.
3. Configuration mismatch — a new environment variable is unset in production, or an existing one is parsed with different semantics, so the code fails at startup or on first use of the config value.
4. Schema-code mismatch — the migration ran ahead of, behind, or differently from the code that expects it, so queries that worked in staging throw in production.

None of these respond to a service restart, which is the reflex fix most teams try first. Restarting an application whose new code cannot parse production config produces exactly one outcome: the same 500, plus the time you spent restarting. The restart instinct is the first thing to override — the second is the debug instinct, covered next.

The First Five Minutes: Triage in the Right Order

The direct answer: the correct order is scope the blast radius, check what changed, decide rollback versus alternatives within 5 minutes, communicate one line, and only then investigate — and every step exists because its inversion is a documented failure mode.

The sequence, with the failure it prevents at each step:

1. Scope the blast radius (60 seconds). Error rate, which service, which routes, which region, is it worsening or stable. One service failing on one route is a regression; three services failing at once is more likely infrastructure — and the difference decides whether rollback is even the right tool.
2. Check what changed (60 seconds). Deploy timeline, config changes, flag flips, certificate rotations, dependency updates in the last 24 hours. The deploy tooling has this answer in one query — kubectl rollout history, the CI deploy log, the flag audit log. Roughly 80 percent of incidents correlate with a change; this step exists to find out if yours is one of them.
3. Make the rollback decision (3 minutes). If a deploy landed inside the window and the error signature is new, the decision matrix in the next section runs in minutes: confidence, migration reversibility, rollback time. The anti-pattern this step prevents is "let me just push a hotfix" — the 3 AM one-line fix that is neither one line nor tested.
4. Communicate one line (30 seconds). "Investigating elevated 500s on orders API, deploy at 13:02 is the suspect, deciding on rollback, update in 10 minutes." Silence breeds escalation; a single status line drops stakeholder anxiety enough to let the responder keep working.
5. Verify recovery, then investigate (ongoing). Error rate below baseline for 10 consecutive minutes before the incident is called mitigated. The postmortem happens after the system is stable, never during.

The order is the content. Teams that investigate before they scope routinely spend 10 minutes reading stack traces for a 2 percent error blip; teams that hotfix before they check the change timeline ship a second defect on top of the first. The five-minute structure is not bureaucracy — it is the compressed version of what experienced responders do by reflex, written down so it survives adrenaline.

The Four Signals That Confirm the Deploy Did It

The direct answer: a deploy is confirmed as the cause when the error signature is new, the start time sits inside the deploy window, the stack trace or error body references the new code, and no other change shares the timeline — each signal is independently checkable in under a minute, and two or more together justify a rollback without further diagnosis.

1. New error signature. The 500s carry an exception type or message your monitoring has never seen. A known, pre-existing error that merely increased in volume is a different incident — capacity, traffic, or a dependency — and rollback will not help it.
2. Start time inside the deploy window. The error-rate curve steps upward within minutes of the deploy completing, and the step, not a gradual ramp. Gradual degradation points at saturation; steps point at changes.
3. Stack trace names the new code. The top frames reference classes, functions, or modules introduced by the release — the fastest single confirmation available, and the one that survives postmortem scrutiny.
4. No competing change. No config push, no certificate rotation, no dependency upgrade, no traffic shift from another team within the same window. A coincidental second change is how teams roll back the wrong thing, see symptoms persist, and end up more confused than before.

Confidence scales with signals, and the matrix in the next section uses that count explicitly. One signal — the deploy overlaps the errors — earns suspicion. Two, including a new error signature or a stack trace that names the new code, earn action. Three or more earn rollback with near-certainty. The expensive mistake is not acting with incomplete information; it is acting in the wrong direction with complete information, which is what happens when the team skips the fourth signal.

Rollback, Partial Revert, or Forward-Fix: The Decision Matrix

The direct answer: roll back when confidence is two or more signals and no irreversible migration shipped in the release; partially revert when the deploy contains a forward-only migration, disabling the new behavior with a flag while leaving the schema in place; forward-fix only when rollback is impossible and the fix is trivial, tested, and certain — and the table exists because choosing the wrong branch of this decision is the single most expensive error in the whole playbook.

Two rows deserve prose. The partial-revert row is the one teams skip and then regret: a deploy that adds a nullable column plus new write logic fails with duplicate-key errors, and the correct move is to flag the write path off and leave the column — harmless, reversible later — rather than reverting the schema. The last row encodes a discipline: one revert at a time, verified, because reverting three changes simultaneously tells you nothing about which one caused the failure, and the next deploy re-introduces two changes that were never the problem.

The Migration-Reversibility Test

The direct answer: before any rollback touches a release that included a schema migration, you must know whether the migration has a working down path, whether it destroyed or renamed existing data, and whether old code can run against the new schema — because a rollback that answers any of these wrongly does not restore the last known-good state, it creates a new, worse one.

Migrations split into reversible and forward-only, and the split is a design decision made long before the incident:

1. Reversible migrations — adding a nullable column, adding a table, adding an index — have a safe down path and old code that runs against the new schema unchanged. Rolling back the code while keeping the schema is harmless; rolling back both is clean.
2. Forward-only migrations — dropping a column, renaming with data loss, splitting a table, rewriting rows in place — cannot be undone without data loss, and old code may not even parse the new schema. These must ship in a release with no other code dependencies on them, so the code can revert independently.

This is the expand-and-contract discipline, and it has converged across tooling: the pattern — expand the schema additively, deploy code that writes both shapes, backfill, switch reads, then contract the old shape in a later release — is now automated in tools like pgroll for PostgreSQL as of its current documentation, and described as a mandatory multi-release process in GitLab's own development guidelines. The operational consequence for the incident: if the deploy followed expand-and-contract, the schema is new-code-tolerant and old-code-tolerant simultaneously, which makes partial revert always available. If it did not, the incident commander needs to know that within the first five minutes, which is why the migration log — what shipped, in which phase, with what down path — belongs in the deploy notification itself, not in a wiki the on-call engineer has to find at 3 AM.

Feature Flags Are the Cheapest Rollback You Have

The direct answer: a new behavior shipped behind a flag can be disabled in seconds without redeploying, which converts the most common class of post-deploy 500 — the new code path failing on production traffic — from a rollback event into a toggle event, provided the flag was actually wired to disable the behavior and not merely to enable it.

The mechanism is simple; the discipline is what fails. A flag that gates new behavior should be independent of the deploy that carried it: the code shipped dark in a previous release, the flag flipped on after the deploy stabilized, and the kill switch tested in staging by flipping it off under load. Feature-flag platforms — Unleash and commercial equivalents — exist precisely because flag state, not code, becomes the rollback lever: toggling a flag off reverts behavior in seconds without a release cycle, per their current product documentation as of October 2026. Three rules make flags an incident tool rather than a liability:

1. The flag must actually kill the feature. A flag that controls entry to the new path but leaves new code executing in shared code paths is not a kill switch — it is a decoration. Test the off state under production-shaped traffic before you trust it.
2. Flag state must be auditable. The incident responder needs to see, in one query, which flags flipped in the last 24 hours — the same timeline visibility the deploy log provides.
3. Flags must decay. Stale flags accumulate until nobody knows which are load-bearing; retire them on a schedule, or the next incident will not know which toggles are safe to flip.

The partial-revert row of the decision matrix is, in practice, a flag operation: the code stays deployed, the flag goes off, the 500s stop. This is why the matrix prefers partial revert for flag-gated releases even when a full rollback would also work — the toggle is faster, carries no schema risk, and preserves the deploy pipeline state for the fixed version that follows.

Making Rollback a 90-Second Operation

The direct answer: rollback speed is a property of release architecture, not of incident skill — blue-green environments switch traffic in one operation, immutable infrastructure reverts to the prior image, and deployment objects with revision history undo in a single command — and the teams that recover in minutes are the teams that paid for one of these architectures in advance.

The three architectures, in increasing order of setup cost:

1. Immutable artifacts with revision history. Every deploy is a versioned image or package; the prior artifact still exists; rollback is redeploying it. Kubernetes expresses this natively — kubectl rollout undo reverts a Deployment to its previous revision, and the command completes in seconds — and every modern CI system that produces immutable artifacts supports the equivalent.
2. Blue-green environments. Two identical production environments; the release deploys to the inactive one, verification happens there, and traffic switches in one operation when it passes. A failure after switchover switches traffic back — the rollback is the same single operation as the deploy, which is the entire point of the pattern. Managed database services have adopted the same shape for schema changes: Amazon RDS blue-green deployments build the new schema on a copy, switch over within a configurable timeout of 30 seconds to one hour per its current documentation, and roll back the switch if the timeout is exceeded.
3. Progressive delivery on top of either. Canary releases and percentage rollouts do not make rollback faster — they make it smaller, by limiting how much traffic a bad release can touch before the error budget trips. A canary that fails at 5 percent is a 5 percent incident; the rollback decision happens automatically before full exposure.

The conclusion from the table, stated plainly: if your rollback takes 30 minutes, the incident is not your on-call's fault — it is the release architecture's bill coming due. Flag-gated releases and immutable artifacts together cover most teams at the lowest cost, and the combination makes the two cheapest rows available on every deploy.

The Second Incident: Rollbacks That Make Things Worse

The direct answer: the three ways a rollback causes a second incident are rolling back across a forward-only migration, rolling back a deploy that was not the cause, and rolling back without a state snapshot of the failing version — and all three are preventable with checks that cost minutes, which is why they belong in the playbook rather than in the postmortem.

1. Rolling back across a forward-only migration. The old code expects a schema shape that no longer exists, or the down migration destroys the data the new code already wrote. The system leaves the 500 state and enters a corruption state — strictly worse, and the reason the migration-reversibility test precedes every rollback decision.
2. Rolling back the wrong change. The deploy was coincidental; the cause was a config push or a dependency outage. The rollback completes, the errors persist, and now the timeline has an extra change in it that the postmortem must unwind. The fourth confirmation signal — no competing change — exists to prevent exactly this.
3. Rolling back without evidence. Reverting destroys the failing state: the stack traces stop arriving, the pods recycle, and the postmortem reconstructs the failure from memory instead of from data. The fix is a snapshot before the revert — logs pulled, a copy of the error output, a copy of the pod spec — so the root-cause analysis happens against the actual failure, not a re-creation of it.

These failure modes are why the rollback decision is a structured 5-minute exercise instead of a reflex. The structure is not there to slow the rollback down; it is there to guarantee that the rollback is the last change the incident needs, rather than the first of several.

Rehearsal: Why Drills Beat Documentation

The direct answer: a rollback procedure that has never been executed under pressure will fail on its first real use — missing permissions, stale runbook commands, a migration log nobody can find — and the only reliable way to surface those gaps is a scheduled drill where someone other than the runbook's author follows it against a deliberately broken deploy in staging.

The drill format that works, and its maintenance schedule:

1. Plant a failure. A staging deploy carries an intentional defect — an unhandled exception, a config mismatch — known only to the drill organizer.
2. Respond cold. An engineer who did not write the runbook follows it verbatim, timing each step. Every command that fails, every dashboard that has moved, every placeholder that was never filled in is a finding.
3. Exercise the flag path and the artifact path. The drill should roll back once via flag toggle and once via artifact redeploy, because the two paths fail differently — flags fail on wiring, artifacts fail on versioning and permissions.
4. Drill quarterly, update after every real incident. Runbooks decay with the infrastructure they describe; the post-incident review owns a runbook update action item, and a CI check can fail alerts that lack a runbook link.

Rehearsal needs an environment you are allowed to break. A staging server with root access, a deliberately planted bad deploy, and enough freedom to roll back across real-looking migrations — that is the entire lab. SurferCloud's ULightHost line covers this use case: root access, dedicated IPv4, and entry configurations from $2.9 per month as of October 2026 per the current promotion page, with the tier list and regional availability on the ULightHost promotions page. A quarter's worth of rollback drills on a machine you own outright costs less than one hour of production incident time, and the drill findings — the broken commands, the missing logs, the unwired flags — are the same findings the real incident would have charged you for at a much higher rate.

Frequently Asked Questions

"My site just started throwing 500 errors right after we deployed — should I debug or roll back?"

Roll back, if the deploy window matches and no forward-only migration shipped in the release. Confirm with two signals — a new error signature and a stack trace naming the new code — then restore the last known-good release. The debugging happens after the system is stable, against the snapshot you captured before reverting.

"How long should I give myself to decide whether to roll back?"

Five minutes. Scope the blast radius, check the change timeline, run the decision matrix, and act. Teams that take longer start debugging instead of deciding, and the debugging path under live traffic is the one that turns a 10-minute regression into a 90-minute incident.

"What if the deploy included a database migration — is rollback still safe?"

Only if the migration is reversible or the old code tolerates the new schema. Adding a nullable column is reversible; dropping, renaming with data loss, or rewriting rows is forward-only. For forward-only migrations, use partial revert — disable the new code path with a feature flag, leave the schema in place — and never roll the schema back across written data.

"The fix looks like one line — why not just fix forward?"

Because "the fix is just one line" is the sentence that precedes prolonged outages. The one-line fix still needs the full release cycle — test, review, deploy — which is 15-45 minutes of customer-visible 500s, and untested emergency fixes carry a real probability of being wrong or incomplete. Roll back now, fix forward properly afterward.

"Can I roll back if I don't use Kubernetes or blue-green?"

Yes, if your artifacts are immutable and versioned. Keep the prior release's package or image available, and rollback is redeploying it — under 5 minutes with a revision history, longer if artifacts are rebuilt from source. Feature flags work at any layer and give you a sub-minute kill switch for new behavior regardless of platform.

"How do I stop my rollback from destroying the evidence?"

Snapshot before you revert: pull the recent logs, copy a failing request's stack trace, save the pod or process spec. The rollback recycles the failing state, and the postmortem needs the real failure — not a reproduction built from memory.

"Should I restart the service first to see if that fixes the 500s?"

No, for post-deploy 500s specifically. A 500 means the process is alive and its code is failing — a restart produces the same 500 plus the restart time. Restarts help crash loops and resource exhaustion, which present as 502s and 504s, not as unhandled exceptions in live code.

"How often should we actually rehearse rollbacks?"

Quarterly at minimum, plus after every real incident. The drill plants a failure, has someone who did not write the runbook follow it cold, and exercises both the flag path and the artifact path — because those two paths fail in different ways, and you want to discover that in staging.