Post Mortem 23rd Jun, 2022 Incident
Post Mortem 23rd Jun, 2022 Incident
On Thursday June 23rd, there was a major outage for cal.com. The reason being some confusion around a script to clone production DB, anonymize it and download for development usage.
On this day, we're trying to test a new service called snaplet.dev that allows us to download our production database, anonymize all private data and use it locally so we can have production-like database for finding rare data cases and fix them.
The setup is pretty straightforward, you connect to your production DB, download it, anonymize it, and restore to your local DB.
The problem here is that everything happened locally in this case. So I've downloaded the data, anonymized it, and when it was time to restore I should have updated the DATABASE_URL to my local one but I've forgot to do it.
The result of this was that I've downloaded production data, anonymized it and uploaded it again to production instead of to my local database. Ending up with mocked data in the production DB.
As many of you might experienced, this ended up having some unexpected behavior like missing bookings, incorrect emails and usernames.
Since my knowledge to access the production backups was limited I was with my hands tied until Bailey Pumfleet gave me the knowledge to actually get to the backups and was able to restore to a previous working version. Although this ended up taking almost 7 hours of outage until it was properly fixed.
As a newly promoted Lead for the engineering team, I have no excuses for this. I should've known better and I can ensure and make sure that this won't happen again.
A big thank you to all the users, customers, team mates and leaders for the patience and trust that has been deposited in me and if given the chance will make sure that this experience is turned into valuable learnings and make us more resilient in the future.
We've fully restored operations and are new checks are in place to prevent this in the future.
Grateful again as always,
Zomars