12 Comments
User's avatar
David Miller's avatar

Thank you for this article; to be honest I didn't know mutation test frameworks even existed. I plugged pitest into my little side project, which I believed had decent coverage. 27% of the mutants survived! So I have work to do.

Expand full comment
Valentina Jemuović's avatar

David, interestingly, that number is similar as what I've observed with teams working on real life projects too. For example, they'd have 80-90% code coverage, but the mutation score may have been just 30%. It's often a big shock when teams see it for the first time, since previously they had false sense of security.

I find mutation testing to be an excellent tool when it comes to Legacy Code, in the following way:

1. First a team should write Acceptance Tests, so that we have protection against user-facing bugs, that's the top priority, must be done before doing anything else.

2. Given the protection of Acceptance Tests, the team can introduce Component Tests, and then Unit Tests. When they get to introducing Unit Tests, that's where I recommend firstly running Code Coverage, getting that up to 100%, and then running Mutation Coverage, score might be low like 20-30% and then getting that up to 100%. Later, when the team shifts from Test Last to TDD, they won't have to rely on Mutation Testing much, it will just be for sanity checking, since with TDD the Mutation Score will naturally be ~ 100%.

Expand full comment
Andrzej Krzywda's avatar

The fun starts when mutation testing exposes alive mutant which are hard to cover with tests. In such case mutation testing shows some design issues.

Expand full comment
Valentina Jemuović's avatar

"which are hard to cover with tests" - could you elaborate further, here ar eyou referring to code that is not exercised by any test...

Expand full comment
Andrzej Krzywda's avatar

When mutant shows alive mutants, usually you try to cover it with a test. But if a test is hard to write (some mocks required for example) then I see mutant as design buddy. It shows me that something is wrong with the design.

It’s similar to normal TDD but here Mutant forces you to do sth about it.

Expand full comment
Jelena Cupac's avatar

Is code coverage a vanity metric without mutation testing? If mutation score exposes untested behavior, why do so many teams stop at code coverage %?

Expand full comment
Valentina Jemuović's avatar

Many teams only know about code coverage, but don't know about mutation testing. They believe that if they have 80% code coverage, that it means 80% of code is protected against bugs. But the reality is it just means that 80% of code is executed through tests, but not protected by tests.

That's why we should use code coverage as the start, but not the end. If we've got low code coverage, we should go and add more tests to at least cover the code. Then it's time to run mutation testing and see whether our tests are actually protecting us against bugs.

Expand full comment
Andrzej Krzywda's avatar

I use mutation testing everyday now. In Ruby we have mutant. Mutation testing is very underrated.

Expand full comment
Valentina Jemuović's avatar

Good to hear that. Which library do you use for mutation testing? Also, do you run mutation tests somewhere on the pipeline...

Expand full comment
Andrzej Krzywda's avatar

In ruby we have a tool called mutant.

Mutant is part of our CI pipeline in this project:

https://github.com/RailsEventStore/ecommerce

we ensure 100% coverage for all domain logic (otherwise it’s risky to refactor)

Expand full comment
Valentina Jemuović's avatar

Thanks, yes, I see Mutation Testing here https://github.com/RailsEventStore/ecommerce/blob/master/.github/workflows/rails_application.yml

In what way do you use mutation testing, do you do Test Last + Mutation Testing, or TDD + Mutation Testing?

Expand full comment
Andrzej Krzywda's avatar

Usually TDD and mutant along the way. Recently I was experimenting with AI doing TDD and mutant (+integration tests) as the guard for agent hallucinations.

Expand full comment