Why E2E Test Initiatives fail

Engineering Managers often try to introduce automated E2E Tests to reduce regression bugs and eliminate manual regression tests. But their initiative fails.

🔒Hello, this is Valentina with a premium issue of the Optivem Journal. I help Engineering Leaders & Senior Software Developers apply TDD in Legacy Code. This article is part of the TDD in Legacy Code series. To get these articles in your inbox every week, subscribe:

---

Initially, Manual Regression Testing seems OK!

When a company is in startup mode, they do everything quickly. The goal is to get customers, get revenue, and get the market position. At this point, all the testing is done manually - maybe just a few hours is enough to do all the manual regression testing. It doesn’t take up too much time. One QA Engineer is enough. Business is happy.

But later, as the product grows, it’s a nightmare!

Then, the company grows. After several years, the software product is bigger and more complex.

The problems faced are:

• Manual Regression Testing takes too long, and even longer

• Developers spend the majority of their sprint bug fixing, not much on features

• You have to keep hiring more QA Engineers, but it’s never enough

The obvious solution is Automated Testing

The “obvious“ solution to the above is to introduce automated testing. Everyone believes they’ll get these benefits:

• Zero regression bugs going into production

• No more manual regression testing by QA Engineers

• Developers will spend most of their time on new features, rather than on bugs

Let’s introduce the E2E Test Initiative

One common approach used by Engineering Managers is to switch manual QA Engineers to Test Automation Engineers, and to transfer manual QA procedures to E2E Tests.

This is common in companies who believe that:

• It’s the QA Engineer’s job to test

• It’s the Developer’s job to code

So then the obvious fix is to optimize the QA Department, which works in silo mode, separated from Developers.

Engineering Managers might try one of the these things:

• Writing E2E Tests that directly use Selenium / Cypress / etc.

• Using some record-and-playback tools to generate E2E Tests

E2E Tests are attractive because they are highly realistic tests, can detect regression bugs in the System and are in communication with External Systems.

Unfortunately, the E2E initiative fails!

Initially, everything looks great, the E2E Tests have eliminated Manual QA Regression Testing.

But then, 3 growing problems occur:

• Unreliable regression bug detection: Initially, the E2E Tests were detecting regression bugs from the end-user perspective, so when an E2E Test failed, developers could fix the regression bug. But later, E2E Tests start failing even though there’s no regression bugs! Then the developers have to waste time troubleshooting E2E Tests, only to discover they’ve wasted time, because there’s no regression bugs!

• High maintenance cost - wasteful: Initially, it didn’t take much time to write the E2E Tests. But later, as the software is changes due to changes in UX/UI design, the E2E Tests start failing even though there’s no regression bug. The QA Engineer has to waste time updating hundreds of tests due to a change in one UI element.

• We can’t test acceptance criteria: We can test many positive flows, but we’re quite limited in terms of negative flows. So, for negative flows, we may still need manual QA testing or we wait for user complaints.

Let’s dive deeper into these problems, and find out how we can solve them: ⬇️⬇️⬇️