73% reduction in Production Bugs @ Carlo Gavazzi Controls (Alessandro Fardin)
Many companies make the mistake - let’s just write Unit Tests, get 80% Code Coverage…
👋 Hello, this is Valentina with the free edition of the Optivem Journal. I help Engineering Leaders & Senior Software Developers apply TDD in Legacy Code.
4000 Acceptance Tests in < 1h
Alessandro Fardin, Software Engineering Team Lead at Carlo Gavazzi Controls, Italy, has shared his success story of using Acceptance Tests on Substack discussion.
“… some years ago we start a Continuous Delivery transformation journey.
Our application is a iiot gateway Linux embedded device
We change lots of things in our behavior, we had only manual tests then we started to add automatic acceptance tests, now we have a suite of 4000 tests that run in parallel in about 30 minutes.”
He summarized the transformation:
“from late phase manual test --> frequent and early manual test
from manual test --> continuous automatic accepting test
continuous automatic accepting test (14 hours)---> fast parallel accepting test (40 minutes)”
Lessons Learnt #1
It’s ok to start with manual testing. Indeed, that’s where most companies are.
Here’s the transition process:
Most companies start with late phase manual testing, typically testing in preparation for a production release. This might delay the release by several months.
The first point of improvement is to switch to earlier and more frequent manual testing, rather than doing it all at the end.
We’ll then start feeling the pain of manual testing, and realize it’s too expensive to do it earlier and more frequently. So then we need to replace Manual QA testing by automated Acceptance Testing.
Lastly, we want to reduce the runtime of automated Acceptance Testing. By parallelizing these tests, it goes a long way forward.
Acceptance Tests in Continuous Delivery
Acceptance Tests are a crucial part of Continuous Delivery, and they have the biggest impact, compared to other lower-level tests.
Many companies make the mistake - let’s just write Unit Tests, get 80% Code Coverage… but yet Acceptance Testing is still done manually - by the QA Engineers.
Fortunately, Alessandro’s team took the right approach, starting top down with Acceptance Tests as a replacement of Manual Regression Testing. This has the highest visible impact - it has the biggest impact in reducing the delivery cycle length AND also reducing production bugs.
Acceptance Testing is crucial in reducing the overall system-level feedback loop and removing the waste of Manual Regression Testing.
Alessandro Fardin shared his story (here's the version translated to English):
What are the results?
Some key ingredients included:
Pipeline
Acceptance Tests
See the Carlo Gavazzi Controls Black Belt Case Study.
The result was a 43% reduction in defects found in production. The feedback cycle was reduced from several months to 40 mins, and deployment from 6 months to 1 month. Developers also reduced time spent on bug fixing by 30%.
Source: Carlo Gavazzi Controls Black Belt Case Study
Lessons Learnt #2
If you’re going to start automated tests, starting with Acceptance Tests. They are the tests with highest visible ROI - testing the system from the End User perspective, testing User Story Acceptance Criteria and it has a direct impact on Bug Tickets.
After you do this, then sure, look at lower level tests.
That’s the same sequence I follow as part of the TDD in Legacy Code series. We start with the System Level first (Acceptance Testing & ATDD) before we go onto apply TDD at the lower levels (Component Testing & CTDD, Unit Testing & TDD).
Share your success story
I’d like to feature TDD success stories in future editions. To share your story, write a comment below. 💪
Want to apply TDD in practice?
You tried TDD, but it didn’t work. That's why I'm going to help you practice TDD step-by-step. Apply TDD on a sandbox project. Access TDD in Legacy Code.
Reducing production bugs by 73% and speeding up deployment from 6 months to 1 month is very impressive Alessandro! Starting with Acceptance Tests is great as they provide the highest visible ROI.
Thank you for posting our story.
It take us a lot of improvements, learnings, and also hard work.
At first, the idea of automating tests and building a CI/CD pipeline can feel overwhelming, even daunting. But you don’t need to start with a perfect or fully-fledged system.
We began with just a few tests and a couple of simple automation scripts. Even these small steps made a big difference: we identified some bugs before they ever reached production.
These early wins gave us (and our managers) the confidence and motivation to keep going. Each new test and every iteration on the pipeline brings us closer to a faster, more stable, and efficient process.
Starting with acceptance tests give us big results, we have now the confidence for improving out unit tests skills and improve our SW architecture
The takeaway? Start simple and watch the value grow. 🚀