Java Magazine, Nov/Dec 2016

ORACLE COM JAVAMAGAZINE NOVEMBER DECEMBER 2016 46 junit 5 until 2009 most academic research looked only at toy projects with less than a hundred lines of code To understand why its so expensive lets look at what a mutation testing tool needs to do Imagine you are performing mutation tests on the Joda Time library which is a small library for handling dates and times This library has about 68000 lines of code and about 70000 lines of test code It takes about 10 seconds to compile the code and about 16 seconds to run the unit tests Now imagine that your mutation test tool introduces a bug on every seventh line of code So youd have about 10000 bugs Each time you change a class to introduce a bug you need to compile the code Perhaps that would take one second So that would be 10000 seconds of compilation time to produce the mutations this is called the generation cost which is more than two and a half hours You also need to run the test suite for each mutant Thats 160000 seconds which is more than 44 hours So performing mutation testing on the Joda Time library would take almost two days Many of the early mutation testing tools worked exactly like this hypothetical tool You still occasionally find people trying to sell tools that work like this but using such a tool is clearly not practical When I became interested in mutation testing I looked at the available open source tools The best one I could find was Jumble Its faster than the simplistic tool I described above but it was still quite slow And it had other problems that made it dificult to use I wondered if I could do better I already had some code that seemed like it might help it was code for running tests in parallel It ran tests in diferent class loaders so that when state that was stored in static variables in legacy code was changed the running tests wouldnt interfere with each other I called it Parallel Isolated Test or PIT After many evenings of experimentation I managed to do better My PIT mutation testing tool could analyze 10000 mutations in the Joda Time library in about three minutes Introducing Pitest My tool has kept the initials of the codebase from which it grew but its now also known as pitest and it is used all over the world Its used for academic research and for some exciting safety critical projects such as for testing the control systems of the Large Hadron Collider at CERN But mainly it is used to help test the kind of nonsafety critical code that most developers produce every day So how does it manage to be so much faster than the earlier system First it copied a trick from Jumble Instead of spending two and a half hours compiling source code pitest modifies bytecode directly This allows it to generate hundreds of thousands of mutants in subsecond time But more importantly it doesnt run all the tests against each mutant Instead it runs only the tests that might kill a mutant To know which tests those might be it uses coverage data The first thing pitest does is collect line coverage data for each test so that it knows which tests execute which lines of code The only tests that could possibly kill a mutant are the ones that exercise the line of code the mutation is on Running any other tests is a waste of time Pitest then uses heuristics to select which of the covering tests to run first If a mutant can be killed by a test Its not possible to perform a mutation test when you have a failing test because doing that would mistakenly appear to kill any mutants that it covered