Today, a Medium article by Benoit Baudry, Professor in Software Technology at KTH and STAMP project coordinator, shares interesting results about Descartes mutation testing tool. This software can automatically short-circuit covered methods and determine a list of pseudo-tested methods in Java projects. Experimenting Descartes over 21 open source Java projects, a total of 28K+ methods could be analyzed, with three main results:
- short circuiting the complete execution of methods provides valuable feedback to developers. The developers have clear goal to write a test: to make this method not pseudo-tested anymore. Developers are more comfortable reasoning at the granularity of a method than at the statement level (fine grained traditional mutation testing).
- short circuiting methods has revealed the presence of pseudo-tested methods in all the projects that we have analyzed, even the ones with very high code coverage. Development teams of all Java projects can benefit from this type of analysis to assess their test suites and improve them.
- interviews with developers reveal that some pseudo-tested methods actually reveal major weaknesses in the test suite. We have collected empirical evidence of test suites fixed after running a short-circuiting experiment.
- For more details, read Descartes article by Benoit Baudry on Medium