Java Magazine, Nov/Dec 2016

ORACLE COM JAVAMAGAZINE NOVEMBER DECEMBER 2016 48 junit 5 Another common problem when setting up pitest for the first time is this message All tests did not pass without mutation when calculating line coverage Mutation testing requires a green suite This message can occur if you have a failing test Its not possible to perform a mutation test when you have a failing test because doing that would mistakenly appear to kill any mutants that it covered Sometimes youll also get the message when all the tests pass when run normally with mvn test If this happens there are a few possible causes Pitest tries to parse the configuration of the Surefire test runner plugin and convert the configuration to options that pitest understands Surefire is the plugin that Maven uses by default to run unit tests Often no configuration is required but sometimes tests need some special configuration in order to work which must be supplied in the pom xml file Unfortunately pitest cant yet convert all the possible types of Surefire configuration If your tests rely on system properties or command line arguments being set you need to specify them again in the pitest configuration Another problem thats more dificult to spot is order dependencies in the tests Pitest runs your tests many times in many diferent sequences but you might have a test that fails if certain other tests run before it For example if you have a test called FooTest that sets a static variable in a class to false and you have another test called BarTest that assumes that the variable is set to true BarTest will pass if it is run before FooTest but fail if it is run afterward By default Surefire runs tests in a random but fixed order The order changes when a new test is added but you might never have run the tests in an order that reveals the dependency When pitest runs the tests the order it uses might reveal the order dependency for the first time Test order dependencies are very hard to spot To avoid them you can make tests defensively set shared state on which they depend to the right value when they start and make them clean up after themselves when they finish But by far the best approach is to avoid having shared mutable state in your program in the first place Finally the setup for using the Google Truth library includes this section excludedClasses param AutoValue_ Expect_ ExpectationFailure param excludedClasses This configuration prevents all classes whose name ends in AutoValue_ Expect_ ExpectationFailure from having mutations seeded into them These classes are autogenerated by the Google Truth build script There is no value in performing mutation testing on them and any mutations that are created would be difficult to understand because you do not have the source code Pitest also provides other ways to exclude code from being mutation tested Details can be found on the pitest website Understanding the Pitest Report Lets do a sample run and look at the result it generates To begin check out the source code for the Google Truth library and run pitest using Maven mvn Ppitest test It should take about 60 seconds once Maven has finished downloading the dependencies After the run youll find an HTML report in the target pitReports directory For the Truth project youll find the report under core target pitReports The pitest report looks very similar to the reports that standard coverage tools produce but it contains some extra