Critique #4 Is Unit Testing Harmful?

Unit Tests can be done "wrong" - resulting in high maintenance cost, risky and expensive refactoring, and even harmful for architecture and design. Unfortunately, this is very common for many teams!

Unit Testing has many benefits.

You’re probably already read about the benefits:

• Regression bug protection during refactoring

• Decreasing development time; increasing productivity

• Improved software design due to testability

The benefits are well-described here Software Engineering at Google: Lessons Learned from Programming Over Time (Chapter 12 Unit Testing):

After preventing bugs, the most important purpose of a test is to improve engineers’ productivity… unit tests have many properties that make them an excellent way to optimize productivity… Due to their many advantages, most tests written at Google are unit tests, and as a rule of thumb, we encourage engineers to aim for a mix of about 80% unit tests and 20% broader-scoped tests.

So does it mean we just read a few tutorials about Unit Testing, buy some courses, and we’re off, ready to go?

Be warned! Danger awaits ahead!

Therein lies the danger, if we do Unit Testing the “good“ way, we’ll get the benefits described above; but if we do Unit Testing the “bad“ way, we’ll be harmed!

Continuing on with Software Engineering at Google: Lessons Learned from Programming Over Time (Chapter 12 Unit Testing - “The Importance of Maintainability“), there’s a classic example of Unit Testing going bad:

Imagine this scenario: Mary wants to add a simple new feature to the product and is able to implement it quickly, perhaps requiring only a couple dozen lines of code. But when she goes to check in her change, she gets a screen full of errors back from the automated testing system. She spends the rest of the day going through these failures one by one. In each case, the change introduced no actual bug, but broke some of the assumptions that the test made about the internal structure of the code, requiring those tests to be updated. Often, she has difficulty figuring out that the tests were trying to do in the first place, and the hacks she adds to fix them make those tests even more difficult to understand in the future. Ultimately, what should have been a quick job ends up taking hours or even days of busywork, killing Mary's productivity and sapping her morale.

Summarized:

1. The developer implements a new feature; the implementation is quick; it requires just a couple dozen lines

2. The developer runs the automated tests and sees so many errors, spending the whole day going through each failure one by one

3. The change that the developer didn’t introduce any actual bugs, yet the tests were failing

Here we see the destructive impact:

Here, testing had the opposite of its intended effect by draining productivity rather than improving it while not meaningfully increasing the quality of the code under test.

Ok, but how frequently does this occur?

This scenario is far too common, and Google engineers struggle with it every day.

So if this is a scenario that Google engineers are struggling with every day, then what about the rest of software engineers across other companies? Indeed, this problem is very common! It’s not surprising that after horrible experiences in unit testing, teams will give up on it as wasteful.

I want to write about the dark reality

It would be too naive to write an article praising Unit Testing when MOST Unit Testing done by MOST developers is harmful. You’ve read more than enough articles saying, “Unit Testing is great, just do Unit Testing, and everything will be amazing!“ That’s a fairytale story.

This article is then dedicated to first understanding the harmful side effects of Unit Testing when it’s done wrong (which happens to be quite often):

• Significantly increasing maintenance costs

• Making refactoring risky and unsafe

• Harming architecture and design

But how could this possibly be? Isn’t Unit Testing suppose to decrease maintenance costs? Aren’t Unit Tests suppose to enable us to refactor safely? Isn’t Testability supposed to lead us to better architecture and design?

You might even say: well, your Unit Tests are bad because you’re not doing it the TDD way. If only you used TDD, then the Unit Tests would “automatically“ be good. That’s yet another misconception that I will address below.

Let’s dig deeper into the all-to-common problem of Unit Testing. Remember the story of Robert, the Engineering Manager who tried “everything“ - he introduced Code Coverage, Mutation Testing, and TDD… They had read books, read tutorials, watched courses, went to some training, and saw FizzBuzz katas. The coverage was great, and the design was testable, yet his team experienced all the problems I mentioned above.

Code Coverage is not the solution.

Mutation Testing is not the solution.

TDD isn’t the solution either…

… if we don’t handle the “elephant in the room“ - the Unit Tests themselves!

The extent to which Unit Tests are coupled to the production code, the higher the maintenance costs, the higher the risk of refactoring, and the worse the impact on architecture and design.

Let’s continue with the story of Robert, the Engineering Manager who introduced Code Coverage, Mutation Testing, and TDD… and then faced disaster due to Unit Tests that were coupled to the structure of the system!

This is a long article (4,000+ words) because this is the BIGGEST PROBLEM I see with Unit Testing (irrespective of TDD).

• Yes, Unit Tests can be wasteful with high maintenance costs - low ROI!

• Yes, Unit Tests can hinder refactoring - contradictory to the TDD cycle!

• Yes, Unit Tests can harm architecture - worsening our designs!

Too shocking!?

I provide arguments below.