High Quality Tests With PIT Mutation Testing

PIT Will Dramatically Improve Your Tests

PIT is a mutation testing tool. It takes your existing unit tests and runs them against a changed version of your code. If your unit tests are thorough, they will catch the change and the test will fail.

PIT is vastly superior to coverage-based testing. Using legacy tools, developers can write tests that achieve 100% coverage, yet test nothing. The most obvious example is a test with no assertions. But even a well-intentioned developer can easily miss testing conditions. Code reviews can help, but a deep code review to get one thorough test class can take an hour. Expanding this to a full codebase is expensive and error-prone.

By contrast, PIT directly checks the quality of the tests. Any line of production code that doesn't have a pass/fail test case is flagged in red. You can set thresholds to automatically check your codebase, and break the builds in your CI tool when the quality falls too low

Easy Integration

PIT has easy maven integration, and great reports. Simply add a few lines to your pom.xml, and invoke mvn org.pitest:pitest-maven:mutationCoverage. The reports will be in your target/pit-reports folder.

Fun To Use

PIT makes improving test quality fun. With colorful reports and numeric goals, improving test quality (and your code) is less of a chore and more of a game.

Encourages Good Coding Practices

Adding new code with PIT in place requires writing thorough tests. Practices that used to be easy to hide, such as copy-and-paste coding, now require extra tests to be written. Code reuse reduces the PIT testing effort, so good coding practices become a win-win for the developer.

Limits

After years of using PIT, I have come to the realization that while it is a dramatic improvement over tools that came before, it is not yet a panacea. PIT's main limit is that it can only check that tests exercise the code that has been written. For code that you have written for certain features or conditions, PIT will ensure it is tested and tested well. However, if there are cases or conditions that you missed or simply skipped, PIT has no way to know. Code that hasn't been written can't be mutated. There is still a need for requirements analysis and code reviews to ensure that all of the required cases are implemented. If your code handles 5 out of 6 conditions, PIT can tell you if your tests check all of the code for those 5 conditions. But it can't tell you that the 6th condition doesn't have any code written for it. So while you may start feeling invincible when your PIT tests reach 100% mutation score, be sure to double-check that all of your requirements are implemented, because PIT can't help with that metric.

Last modified on 1 May 2020 by AO