How do you know if Unit Tests are protecting you against regression bugs? Code Coverage can't help, because it checks only whether you executed code, but not whether you verified behavior of the code.
Thank you for this article; to be honest I didn't know mutation test frameworks even existed. I plugged pitest into my little side project, which I believed had decent coverage. 27% of the mutants survived! So I have work to do.
David, interestingly, that number is similar as what I've observed with teams working on real life projects too. For example, they'd have 80-90% code coverage, but the mutation score may have been just 30%. It's often a big shock when teams see it for the first time, since previously they had false sense of security.
I find mutation testing to be an excellent tool when it comes to Legacy Code, in the following way:
1. First a team should write Acceptance Tests, so that we have protection against user-facing bugs, that's the top priority, must be done before doing anything else.
2. Given the protection of Acceptance Tests, the team can introduce Component Tests, and then Unit Tests. When they get to introducing Unit Tests, that's where I recommend firstly running Code Coverage, getting that up to 100%, and then running Mutation Coverage, score might be low like 20-30% and then getting that up to 100%. Later, when the team shifts from Test Last to TDD, they won't have to rely on Mutation Testing much, it will just be for sanity checking, since with TDD the Mutation Score will naturally be ~ 100%.
When mutant shows alive mutants, usually you try to cover it with a test. But if a test is hard to write (some mocks required for example) then I see mutant as design buddy. It shows me that something is wrong with the design.
It’s similar to normal TDD but here Mutant forces you to do sth about it.
Many teams only know about code coverage, but don't know about mutation testing. They believe that if they have 80% code coverage, that it means 80% of code is protected against bugs. But the reality is it just means that 80% of code is executed through tests, but not protected by tests.
That's why we should use code coverage as the start, but not the end. If we've got low code coverage, we should go and add more tests to at least cover the code. Then it's time to run mutation testing and see whether our tests are actually protecting us against bugs.
Usually TDD and mutant along the way. Recently I was experimenting with AI doing TDD and mutant (+integration tests) as the guard for agent hallucinations.
Thank you for this article; to be honest I didn't know mutation test frameworks even existed. I plugged pitest into my little side project, which I believed had decent coverage. 27% of the mutants survived! So I have work to do.
David, interestingly, that number is similar as what I've observed with teams working on real life projects too. For example, they'd have 80-90% code coverage, but the mutation score may have been just 30%. It's often a big shock when teams see it for the first time, since previously they had false sense of security.
I find mutation testing to be an excellent tool when it comes to Legacy Code, in the following way:
1. First a team should write Acceptance Tests, so that we have protection against user-facing bugs, that's the top priority, must be done before doing anything else.
2. Given the protection of Acceptance Tests, the team can introduce Component Tests, and then Unit Tests. When they get to introducing Unit Tests, that's where I recommend firstly running Code Coverage, getting that up to 100%, and then running Mutation Coverage, score might be low like 20-30% and then getting that up to 100%. Later, when the team shifts from Test Last to TDD, they won't have to rely on Mutation Testing much, it will just be for sanity checking, since with TDD the Mutation Score will naturally be ~ 100%.
The fun starts when mutation testing exposes alive mutant which are hard to cover with tests. In such case mutation testing shows some design issues.
"which are hard to cover with tests" - could you elaborate further, here are you referring to code that is not exercised by any test...
When mutant shows alive mutants, usually you try to cover it with a test. But if a test is hard to write (some mocks required for example) then I see mutant as design buddy. It shows me that something is wrong with the design.
It’s similar to normal TDD but here Mutant forces you to do sth about it.
Not sure if I got it. Perhaps you can provide an example (can be in pseudocode)
Is code coverage a vanity metric without mutation testing? If mutation score exposes untested behavior, why do so many teams stop at code coverage %?
Many teams only know about code coverage, but don't know about mutation testing. They believe that if they have 80% code coverage, that it means 80% of code is protected against bugs. But the reality is it just means that 80% of code is executed through tests, but not protected by tests.
That's why we should use code coverage as the start, but not the end. If we've got low code coverage, we should go and add more tests to at least cover the code. Then it's time to run mutation testing and see whether our tests are actually protecting us against bugs.
I use mutation testing everyday now. In Ruby we have mutant. Mutation testing is very underrated.
Good to hear that. Which library do you use for mutation testing? Also, do you run mutation tests somewhere on the pipeline...
In ruby we have a tool called mutant.
Mutant is part of our CI pipeline in this project:
https://github.com/RailsEventStore/ecommerce
we ensure 100% coverage for all domain logic (otherwise it’s risky to refactor)
Thanks, yes, I see Mutation Testing here https://github.com/RailsEventStore/ecommerce/blob/master/.github/workflows/rails_application.yml
In what way do you use mutation testing, do you do Test Last + Mutation Testing, or TDD + Mutation Testing?
Usually TDD and mutant along the way. Recently I was experimenting with AI doing TDD and mutant (+integration tests) as the guard for agent hallucinations.
What did you learn during that experiment?