Release Failures Hit Zero. Then They Bounced Back

A diagnosis, a countermeasure, and one real case. Four months of release data, and the month the gains slipped back.

The number had just hit zero.

For the first time in months, the number of release failures reaching production was zero. Three months of digging had gotten the team there. I could have written up the win that same afternoon and gone home.

One month later, it was back up.

I have watched this shape play out on enough teams to trust it. A team fixes the real thing, the number falls, everyone breathes out, and then it drifts back. Here is the story of one team that lived exactly that, and the part that finally held.

Where the failures came from

The team was tracking one number, month by month: release failures that reached production. In the first month I was there, it sat at its high point. Let’s call that the baseline. A release failure here means a change that had to be applied again because the first attempt broke something. For example, a piece of code that would not compile, a script that failed, or a step that was wrong.

Then it started to fall. By the second month, it was under a fifth of the baseline. By the third, zero. The number showed the team started fixing things.

Under that number sat a second one nobody watched as closely. Over that same stretch, fourteen releases failed in the lower environments before they ever reached production. Six in the acceptance environment, the rest spread across three integration environments. Those failures were the early warning. The impact line in the analysis said it plainly: more failures below meant more risk above.

So when the production number climbed back in the fourth month, I sat with the team and we ran a root cause analysis on the confirmed failures. It was not a blame round. For each failure, we clarified what broke, why, and whether the dev team confirmed it by observation.

More than half traced to one thing. A database upgrade that had fallen behind. Some environments had it, one did not, and the same code passed where the upgrade was done and failed where it was behind.

The rest came from somewhere else. None of them was a developer being careless. Each of them was a gap that no single person owned.

Why “roll it back and move on” doesn’t work

Search how to handle a run of release failures, and you will see the advice converge fast. Roll back to the last good version. Tell the users. Automate the pipeline so people touch it less. Add health checks and gates that stop a bad build. Put the change failure rate on a dashboard and watch it fall. Run a post-mortem on the ones that hurt.

Most of it is sound. It still would not have found this problem.

It fails because it does not go beyond the fix that gets production working again. Recompile the tag, rerun the script. The service is restored, and the cause untouched. The next release stays exposed.

Every one of those standard moves makes survival faster: roll back sooner, detect earlier, shrink the blast radius. All of it is worth having. None of it asks why the failure was born. A rollback turns production green again and says nothing about the version gap between the two environments that caused the code break. A dashboard tells you the release failed. It will not tell you the same code compiled fine one environment over.

The post-mortem gets closer, and the good teams run one. But most run it only on the failures that reach production and hurt someone, then stop once the incident is closed. The failures on this team were not all in production. Fourteen of them sat in the release path, across four environments, and never reached a user. You do not find those by watching production only. You find them by watching the whole path and treating every failure as a question rather than focusing only on the loud ones.

Build the root-cause reflex

The fix was a habit, not a tool. Take every failure back to its root, and to do that, watch the entire release path rather than focusing just on the end.

So the team lead sat with the confirmed failures, one at a time, and refused to stop at the symptom. “The tag failed” is a symptom. The real work was the next question down. Why.

The first tag would not compile. Why? It used database functions that existed only in the newer version. Why did that matter? One environment’s upgrade had fallen behind, so the same code met two different databases. The cause was not the tag. It was version drift across environments.

The second failed on a variable declared in a form that the older version did not accept. It is the same family: code written for one version, run against another.

The third was a grant command with the wrong syntax in a table script. Trace it down, and the cause was a script that was never validated against the environment it would actually run in.

The fourth ran against incorrect information during a post-install step. The cause: the deployment instructions were trusted and never checked against reality.

Four failures, and under them two causes: the test and production environments were on different database versions, and some release work shipped without a check against where it would run. Trace those down one more step, and they meet in the same place. Nobody owned the release path between environments, so nothing verified that the environments matched, and nothing validated the work against its target. That gap was the root. You see it only when you line the failures up and trace each one past the fix, down to the reason. Every cause was confirmed through observation with the dev team before anyone built a countermeasure for it. There was no guessing.

Out of that analysis came the changes that stuck.

Fixed the drift at the source. The late upgrade got done, and code that needed the newer version was compiled and validated against the version it would actually run on.

Front-loaded the slow work. Anything with a known wait, an upgrade or an access right, now gets requested at the start of a deployment. It runs in parallel instead of blocking the release at the end.

Put the whole path under watch. Every environment, not just production. A failure below production now counts as signal, not noise, which is how the fourteen quiet ones became visible in the first place.

Made the analysis mandatory. In any month when the failure count is not zero, the team systematically conducts root cause analysis on all issues.

Watch a professional tennis player return a serve. They do not decide to move. The reflex fires before the conscious thought, trained by ten thousand repetitions until it runs on its own. Root cause analysis is the same muscle. The first one this team ran was slow, deliberate, and a little awkward. By the fifth, they were tracing a failure to its root without being asked. That reflex is the countermeasure. The tracking, the rule, the front-loading, all of it is scaffolding around the one thing that matters: a team that meets a failure with “why,” automatically.

Four months later

Here is what the four months looked like.

The first month is the baseline. By month two, the team had cleared more than 80% of the failures. By month three, all of them. The reduction read 100%. If we had stopped then, this would have looked like a clean win.

Month four handed half of it back.

The reduction fell from 100% to 50%. Not all the way, but far enough to count as a miss again. The gain did not hold on its own.

Here is the part most case studies cut. A number that touches zero once is not a system that holds zero. The month-three zero was real. It was also fragile. The moment a fresh batch of changes went out against the same version gaps, some of them came back as failures.

But the reflex held. A miss meant the dev team ran the analysis, and they did. That fourth-month spike is not the story failing. It is the story working, because the spike triggered exactly the response it was built to trigger.

50% removed and holding by the fourth month, with a peak in the middle, and a slip that taught the team more than the peak ever could. Improvement is not a line sliding down. It is a number that climbs back, a team that runs the analysis again, and a countermeasure that holds a little better each pass.

The pattern

Engineering teams don’t have a release problem. They have a system that lets environments drift apart, then hides the drift until a release fails on top of it.

That team had no AI anywhere near its toolchain. Yours does. AI does not know your environments are out of sync. It writes against whatever version it is handed and uses the newest feature it can, whether or not the environment next door can run it. It does not just send more releases at the gap between your environments. It widens the gap faster than a human would, and the failure still shows up late, when the date is already close.

There is a harder truth, and the fourth-month bounce carries it. Fixing a number is easier than holding it. You can cut release failures in a quarter. Keeping them down is a different job, and it’s the team’s responsibility. The tracking and the rule are what you set up in an afternoon. The reflex that meets every failure with “why” is what takes months, and it is the only thing that keeps the number down after I leave.

Trace the failures to their root first. The releases follow.

If you suspect your release failures are hiding between your environments, start with the Delivery Scorecard. Two minutes, ten questions, and you know where to look first. And when AI tooling is part of the picture, I run a half-day working session with leadership teams on exactly this. Reply to this email, or message me on Substack, and I’ll send you the outline.

This story is based on a real coaching project. Some details have been changed to ensure confidentiality.