Wednesday, 29 June 2016

Don't forget about rule #1

So I screwed up...

Yes, as hard as that is for me to say. I royally screwed up (with some help from a colleague).
When setting up a new live deployment of your product. ALWAYS SET UP BACKUPS FOR YOUR DB's. Yes, it seems stupid that I should even say this, but for the first time during my 8 year career, I forgot to set up backups. The product went live about a week before and a colleague accidentally restored our blank database backup over the production database... Lots of data loss!


  • NOW WHAT?
  • WHERE DO I START
  • HOW DO I TELL THE CLIENT
  • CAN WE RECOVER?
I decided to Google How to roll back a database restore. You have to try right. No luck. I got served up countless posts about people in the same situation, all being told: "Restore your latest full backup and then restore the balance from your transaction log backup." Great... And then a few unlucky souls that ended up in the same situation with no options. Oops.
So after this, I had to call the client and explain what happened. "Can't you just restore the backups?" I got asked. Uhm... No, we forgot to set up backups.
In this instance, luckily the product only went live a couple of hours before, so there weren't that many transaction we had to recapture (Our point of sale prints out physical slips, we manually recaptured transactions from slips...) The major headache was getting everything set up again, basic configurations, logos, company and customer information. This manual fixing took about 2 hours and took about 2 years off my life expectancy. This was followed by waiting to receive scans of the slips to recapture transactions manually.
On the up side, the client was really impressed by how quickly we got things up and running again. A small consolation for the stress and the neck spasm I developed during the 2 hour ordeal.
To prevent this from happening again, we summarily ran through all of our systems including demo systems and test systems and ensured the backups were in place and running. We identified another issue, we had our demo databases running along side our production databases, way too easy to get access to the production databases then. We need to fix that! We updated our documentation with the steps to set up a new installation to include in big bold letters Set up backups. Lastly we will discuss this during our next retrospective to find solutions and implement safeguards such as restricting access to production databases and making sure that we work with a peer when doing things like this to make sure we don't miss out any crucial steps. 

Perhaps a new found appreciation for peer coding from my side. Maybe Extreme programming (XP) is onto something with always working in pairs.