Many companies try E2E tests and Unit Test initiatives.
The might write 10 tests. 100 tests… 1000 tests.
… But did NOT achieve the goal of reducing production regression bugs.
Management is getting angry - why is time being spent on test automation, without any results to show for it?
They are right.
If we don’t achieve any outcomes, then the test automation is a WASTE of time.
Why E2E Tests don’t protect us in production
Limited scenarios
We can cover some scenarios:
Testing that given valid inputs, we can place an order
But there’s many scenarios that we can’t cover because we can’t control the external systems:
Can’t test total order total price, because we can’t control the ERP product pricing
Can’t test that error popup when payment system is down, because we can’t force PayPal to go down
Can’t test order discount when order is made on weekend after 5pm, because we can’t control the system clock
Maintenance issues
Furthermore, E2E tests are flaky and hard to maintain. If you change one thing in the UI, this can break hundreds of tests, and then you have to fix all of them. Teams often spend more time maintaining tests than actually delivering new features.
Why Unit Tests don’t protect us
Unit tests don’t protect us against regression bugs because they cover only Business Logic. They don’t cover I/O and Presentation Logic, and they don’t cover the whole user flow. That’s why you can have hundreds of thousands of tests passing, 100% code coverage… but coverage is only for business code logic, not infrastructure or presentation logic.
Why Acceptance Tests CAN protect us
Since those two approaches don’t work, what does work is Acceptance Tests.
Acceptance tests are like E2E tests but without the limitations of E2E tests.
