Your Rollback Succeeded. Production Hasn't Recovered Yet.
A production incident changed how I think about rollbacks, recovery, reproducible artifacts, and the difference between restoring software and restoring the business.
A while ago, I was pulled into a P1 because a critical order-processing workflow had stopped processing orders.
A new feature had recently gone to production.
We found a problem in the new validation logic, decided a rollback was safer than a hotfix, rolled it back, and the deployment succeeded.
Green pipeline.
Feature reverted.
Incident resolved.
Except it wasn't.
A few hours later, Production Support came back.
Orders still weren't making it through.
That incident changed how I think about production recovery.
I had treated a successful rollback as evidence that the system had recovered.
It isn't.
A deployment is an engineering event. Recovery is a business state.
Those are not the same thing.
Start with what changed
When Production Support first reported that the order-processing workflow was failing, I started with one of the simplest questions in production debugging:
What changed?
This sounds obvious, but during an incident it's surprisingly easy to skip.
People are messaging you.
Production Support wants updates.
Someone is checking logs.
Someone else has a theory.
Another person remembers an incident from six months ago that was "kind of similar."
Meanwhile, chat notifications are doing their best impression of a distributed denial-of-service attack against your attention.
My first job was to keep myself — and everyone around me — calm enough to investigate systematically.
I told the team I had started debugging and checked the release documentation for that week.
Several components had been deployed as part of a new feature in the order-processing workflow.
The workflow was orchestrated through multiple serverless functions, but only two had changed as part of the feature.
That reduced the search space.
I checked their logs.
Nothing obvious.
I checked the downstream service.
Nothing obvious there either.
Ah, production debugging.
Everything is broken.
According to the logs, everything is having a wonderful day.
So I stopped looking only for exceptions and started tracing what the new feature actually changed.
Part of the new validation logic fetched information from a data store.
Production contained duplicate records that the new logic hadn't accounted for correctly. The application wasn't consistently selecting the expected information, and incorrect data was flowing into the order-processing workflow.
Now I had a decision to make.
Fix forward?
Or roll back?
Rollback versus hotfix is a risk calculation
I don't believe "always rollback during a P1" is particularly useful advice.
Sometimes rollback is safer.
Sometimes it isn't.
Suppose the production bug is a small conditional error, the correct behavior is well understood, and rolling back requires coordinating five independently deployed services.
A focused hotfix might carry less risk.
Now change the scenario.
The failure involves unexpected production data. The correct fix isn't obvious. Validating the fix requires regression testing rather than a simple smoke test. Meanwhile, a critical business workflow is unavailable.
That's a very different calculation.
For me, the decision comes down to questions like:
- How well do we understand the failure?
- How large is the proposed fix?
- What testing does the fix require?
- How large is the rollback blast radius?
- How quickly can each option restore the business capability?
- What happens to the business while we're deciding?
In this incident, fixing the duplicate-data behavior correctly wasn't straightforward.
Time mattered.
I recommended rolling back the new validation feature and restoring the previously working behavior.
That should have been the boring option.
Production had other plans.
A rollback target is more than a Git commit
We attempted the rollback.
The CI/CD pipeline failed.
The application hadn't changed since that previous version ran successfully.
But something around it had.
The package registry infrastructure had changed. The repository configuration had been updated for the new registry, but the existing dependency lock file still contained references associated with the previous package infrastructure.
The old source code still existed.
The environment required to build it exactly as before did not.
That's an important distinction.
We often talk about rollback as though we're doing this:
Current Version
↓
Previous Version
But "previous version" is rarely just source code.
What actually executes in production is closer to:
Source Code
+
Dependency Graph
+
Configuration
+
Build Environment
+
Runtime
+
Infrastructure Assumptions
If any of those have changed, going back to an old commit does not necessarily mean you're restoring the exact system that previously worked.
I needed to make the rollback deployable through the current package infrastructure.
So I regenerated the dependency lock file.
The dependencies resolved correctly.
The pipeline ran.
The deployment succeeded.
Green.
Beautiful.
I told Production Support that the validation feature had been rolled back and we should be good.
There's a sentence every production incident eventually teaches you to fear:
"We should be good."
Narrator: We were not good.
The recovery path can introduce its own failure
A few hours later, Production Support reported that orders still weren't coming through.
My first thought was that the workflow crossed team boundaries.
Multiple teams owned different pieces of the overall flow.
I had rolled back my side.
Had something changed elsewhere that also needed to be reverted?
I reached out to the other team.
Nothing had changed there.
No rollback was necessary.
That's when the alarm went off in my head.
What had I changed during recovery that wasn't part of the original application behavior?
The dependency lock file.
Back to the serverless function.
The existing logs showed that it was failing, but not why it was failing.
So I added more logging.
The additional telemetry finally exposed the issue.
Regenerating the dependency lock file had changed a library involved in making an HTTP call to a downstream service. The newer version no longer behaved correctly with the import pattern the function was using.
The original incident had been caused by the new validation behavior.
The recovery attempt had introduced a second failure through dependency drift.
That's the part of the incident I find most interesting now.
Recovery procedures are production changes too.
They deserve the same skepticism we apply to ordinary deployments.
A rollback is not automatically safe because its intention is to restore safety.
"Known good" needs a stronger definition
Engineers often use the phrase known-good version.
I used it too.
But after this incident, I think that phrase needs qualification.
What exactly is known good?
The source code?
The compiled artifact?
The container?
The dependencies?
The configuration?
The infrastructure it ran against?
If performing a rollback requires rebuilding an old version against today's environment, I'm not necessarily deploying yesterday's known-good artifact.
I may be creating a new artifact from old source code.
Those are not equivalent.
In this incident, regenerating the dependency lock file was necessary to get the old code through the current build infrastructure.
But doing so changed the dependency graph.
The source code had gone backward.
Part of the runtime behavior had quietly gone forward.
That combination created a failure that had not existed before the recovery attempt.
Also, package managers are dangerous.
I'm mostly joking.
Mostly.
A green deployment proves surprisingly little
After fixing the dependency problem, I deployed again.
The deployment succeeded.
But this time I didn't declare victory.
I went through an actual customer-facing entry point and submitted an order myself.
The order successfully traveled through the workflow.
Then I asked Production Support to replay the orders that had failed during the incident.
Those processed successfully too.
That was recovery.
Looking back, I think my mental model during the first rollback looked like this:
Incident
↓
Root Cause Narrowed
↓
Rollback
↓
Deployment Successful
↓
RECOVERED
The incident taught me that the model was incomplete.
A better model is:
Incident Detected
↓
Failure Contained
↓
Software Restored
↓
Critical Workflow Validated
↓
Failed Work Reconciled
↓
Business Capability Restored
↓
RECOVERED
The CI/CD pipeline can tell me whether my deployment succeeded.
It cannot tell me whether customers can successfully place orders again.
Those are different assertions.
Recovery should be defined by business invariants
This is probably the biggest thing I took away from the incident.
During recovery, we tend to validate technical state.
Is the function healthy?
Is the deployment green?
Are there exceptions?
Is the service responding?
Those signals matter.
But they are proxies for what we actually care about.
For this system, the meaningful questions were much simpler:
Can a new order successfully make it through the critical workflow?
And:
Can the orders that failed during the incident be recovered?
Until both were true, the business capability had not actually been restored.
This changes how I think about incident validation.
For a payment platform, the recovery invariant might be:
Can a customer successfully complete a payment?
For a shipping platform:
Can a shipment successfully purchase a label?
For an integration platform:
Can an event travel from publisher to subscriber and be acknowledged?
The exact invariant depends on the system.
But every critical workflow should have one.
If your incident playbook ends with:
Deployment successful.
I think it's missing a step.
Observability should follow the workflow
There was another pattern in this incident that I've encountered more than once.
The logs weren't sufficient when I needed them most.
I could see that a function failed.
I couldn't see why.
Technically, that's observability.
Practically, it's not very useful observability.
Critical workflows should make it possible to reconstruct the journey of a business transaction through the system.
For an order, I want enough information to answer:
- Where did the order enter the system?
- Which components processed it?
- What decisions were made?
- Which external calls happened?
- Where did it stop?
- Can I correlate the same order across service boundaries?
You don't discover whether your logging strategy answers those questions when everything is healthy.
You discover it during a P1.
Preferably, you discover it before then.
Production data has history
The original validation failure contained another lesson.
The feature encountered duplicate records that weren't sufficiently represented during testing.
Production data has history.
It has migrations.
It has old assumptions.
It has weird records.
It has edge cases created by versions of the system nobody remembers anymore.
QA data often has excellent manners.
Production data does not.
For critical workflows, testing shouldn't only validate the expected data model.
It should validate the actual shapes and inconsistencies the system may encounter.
That doesn't mean copying sensitive production data into lower environments.
It means understanding production characteristics well enough to represent them safely in test scenarios.
If your architecture assumes:
There will always be exactly one record.
then "exactly one record" isn't merely a data assumption.
It's a business invariant.
Either enforce it or design for what happens when it isn't true.
Production will eventually ask.
What I do differently now
This incident changed several things about how I approach production recovery.
After a rollback or hotfix, I validate the actual business workflow instead of stopping at deployment success.
If the scope of a feature is large and I'm not familiar with all of its behavior, I reach out to the product or domain experts rather than filling gaps with assumptions.
I involve the appropriate teams in smoke testing critical workflows.
I pay more attention to dependency artifacts when build infrastructure or package registries change.
And I want critical workflows to have enough telemetry that adding logs isn't a prerequisite for understanding why they're failing.
But the biggest change is how I define done during an incident.
A green pipeline isn't done.
Healthy infrastructure isn't done.
A successful rollback isn't done.
Even fixing the root cause isn't necessarily done if failed work is still sitting somewhere waiting to be recovered.
The incident is done when the business capability is restored.
So after a production fix, I try not to ask:
Did the deployment succeed?
I ask:
Did the workflow succeed?
And if there was a backlog:
Did the system recover what failed while it was broken?
Those questions have saved me from trusting a very convincing green checkmark ever since.
System Note
Principle
Production recovery is a business state, not a deployment event.
A useful recovery model
Failure contained
↓
Software restored
↓
Critical workflow validated
↓
Failed work reconciled
↓
Business capability restored
A rollback isn't necessarily the old system
If an old version must be rebuilt against changed dependencies, configuration, build infrastructure, or runtime assumptions, treat the resulting artifact as a new production change and validate it accordingly.
And finally
If you regenerate a dependency lock file during a P1, maybe take a very close look at what changed.
Ask me how I know.