bug in production

I Pushed a Bug in Production

Reading Time 3 minutes

It’s inevitable, at some point you will push a bug in production

Statistically speaking, at some point you will push a bug to production. It may be something small, it may be something big. It might be a small UI issue, it may be a rather large functional issue. Sadly, at some point it will happen.

So, how can we stop this, how can this be mitigated?

Stop Bugs going in Production

Unknowingly, I have pushed bugs to productions. Not my most favorite confession but it has happened. It wasn’t the most joyful of feeling, but you live and learn.

Here is a list of actions I live by to help reduce instances of bug getting past me, into production.

Play ‘The 5 WHY’ Game

This is perhaps one of my most powerful techniques. The ‘5 WHY’ approach is simple, you ask yourself 5 WHY based questions which help to diagnose the root cause to a problem.

For instance, let’s assume a bug around user login made it into production.

  • 1. WHY was the user not logged in?
  • 2. WHY was the user account not recognized?
  • 3. WHY did the user account have a duplicate user ID?
  • 4. WHY was the database able to store multiple user accounts with the same ID?
  • 5. WHY was the ID code generator not producing unique ID’s?

Following the above line of question would help to identify a flaw in the process. Perhaps if there was an automated test for ID generation, the bug may not have gone through to production.

The ‘5 WHY’ game has helped me to think about bug prevention in production. It has also helped me to triage situations where a bug has gone through.

Catch It, Automate It

Not a super fan about this particular approach but one of the best ways to ensure that the same bug does not make it through again is to write an automated test for it. This approach may however unnecessarily balloon the number of tests that you have. You may write many tests which run every time but not add much value since they are trying to capture something very specific.

On the other hand if we don’t write a test for it, it may go through again. That’s something no one would want on their conscious.

Big Bang Integration

Test your code with everyone’s code. Do not merge directly to some master code base, merge to a release code base and test it. If your release code works, put it in master.

A very simple approach to stopping bugs going to production is to pretend it did not happen, one way to achieve this is with multiple code bases. At any point in time, the production code that is running will be the latest version of master. In an ideal world, the master version is deemed stable and functionally working. When pushing new code to production, it is better to push a release version instead.

Pushing a release version somewhat reduces bugs in production. If the bug is found, you can quickly change back to a version of your code which is deemed better.

Process Failure, Not People

The biggest and most important thing to identify is this:

Bugs in production is not a result of a person’s failure, its the fault of a Process

Sadly at some point, most likely during a blue moon, you will push a bug. You are human, you will make mistakes or more likely you will miss things.

It’s is far better (and perhaps more healthy) to remedy ‘preventing bugs in production’ by putting in place a process which can help to prevent it.

Processes may include:

  • Writing automated tests
  • CI jobs and pipelines
  • Release processes
  • In closing, in an ever growing Software Industry, trying to prevent bugs in production is pretty much impossible.

    But we can give it our best.

    Mo

    I am a passionate tester, father, husband and gamer. I love to write blogs about Software Testing and generally contribute back to the Software Testing world.

    More Posts - Twitter - Facebook

Published by

Mo

I am a passionate tester, father, husband and gamer. I love to write blogs about Software Testing and generally contribute back to the Software Testing world.