Production Nightmare

How can you prevent accidental changes in production?

👋 Hello, this is Valentina with the free edition of the Optivem Journal. I help Engineering Leaders & Senior Software Developers apply TDD in Legacy Code.

---

The production database connection string was in the documentation. There were instructions to replace it with the local one, so you had to be really careful. This was assumed to be common knowledge.

(Or, in some companies the documentation isn’t clear - it just says “check your database connection string” with some details, but doesn’t explain what the production string looks like. When a new developers comes in, they might not even realise that the connection string they’re using is linked to production.)

Gary made a simple mistake

Gary - the new developer was following the setup procedure. But he didn't replace the database connection string with the local one. He accidentally forgot this step.

So, when the app started up, he ended up coding and changing stuff in production - this was discovered only when the customer support calls started coming in.

Gary needed to do some local testing, so he decided to delete some orders - not knowing that he was in production. Customer support began receiving calls from users that their orders are gone and they didn't get the deliveries.

This incident got escalated to the CTO, who then informed the Engineering Manager. When the Engineering Manager confronted the team, they discovered that Gary was deleting data in production. The Engineering Manager blamed the developer: “Why didn’t you read the instructions? We can’t trust you - you made such a huge mistake.”

But the Team Lead, who had been aware of this problem for a long time, saw that it wasn’t the developer’s fault - it was a problem with their process.

The accidental commit

Digging deeper, the Tech Lead found another related issue: previously, a developer had been debugging in production from their local computer. To do that, they edited the configuration file on their local machine to include the production database connection string. Normally, they would revert that file back to the local database connection afterward - but this time, they forgot to do it.

So, the configuration file still pointed to production, and when they committed their changes, they accidentally committed the config file connecting to the production database.

The solution: Isolating environments

So, they ended up switching off access to the production database. It was no longer possible to connect to production from a local environment.

---