Practicing effective mutation testing
How do you see mutation testing tools being adopted in business projects, as an effective method with significant benefits on software updates?
The most effective way I've seen mutation testing tools being deployed in business projects is when a developer simply starts using one locally to check their own work as they develop. If they are self motivated to write good code and good tests then the tool saves them time and effort by automating some of the thinking and highlighting areas of the code and tests that need attention.
I am often contacted by people trying a top down approach where mutation testing is setup on a CI server and developers then told that they must attain some mutation score or other. I've never tried this with mutation testing, but have worked in situations where this has been done with plain code coverage. The results were never good, the code coverage scores went up but the benefit that these score is meant to be a proxy for (good tests that enabled the code to be refactored) was never achieved. Instead developers generally wrote tests that were either ineffective or, much worse, typely coupled and overfitted to the code making it hard to change.
Can you give us more information about today’s business challenges solved by Pitest? How is it relevant in today's environment compared to 10 years ago?
The challenge is "how can we push out code that works and is easy to change, in order to enable our business of doing X".
A key learning in industry over the last 20 years or so is that simple code with well written tests and quick feedback cycles, is easier to change than "clever" code without tests or with slow feedback cycles. Pitest automates checking one aspect of a good test (its strength), so allows code to be developed more quickly with fewer mistakes.
Unfortunately strength is only one aspect of what makes a test good so, like any tool, the person wielding needs skill and understanding about what it is they are trying to achieve.
I don't think the relevance has changed greatly compared to 10 years ago. I often see it asserted that mutation testing tools are more relevant now as we have access to more computational power, but I am not sure this has been the main driving factor. Looking at the tools that are in common use (pitest, mutant, infection) the main change has been along the lines of "do-smarter" - making more effective use of the available CPU cycles. It is telling that languages such as C# where tools that implement the key coverage targeting optimisation have not been available have seen little practical uptake of mutation testing, despite the equal availability of more computational power.
What kind of sector in the industry is using more Pitest? And how frequently?
I have a small insight into this based on hits to the pitest website from links hosted on corporate CI servers. Based on this the main users would appear to be the financial services and insurance industry, but this sample is skewed towards people running pitest in a certain way.
Based on contacts via e-mail, pitest is being used in a very wide variety of industries (recruitment, bio tech, fashion, tractor sales, big science). It is not possible to say how frequently, personally I run it against code I'm working on upto a few hundred times a day.
Do you have to convince new DevOps teams that mutation testing improves the code quality? Is that a cultural challenge?
I don't try to convince people to use mutation testing. If a team needs convincing there are probably more important things they need to do before they start using a mutation testing tool (learning how to write "good" tests being the most important one - where "good" covers much more than the test's ability to detect faults).
There are an infinite range of different attitudes and cultures in development ranging from "tests are a waste of time" through to "I see no point in mutation testing because I'm a rockstar developer and there can't possibly be anything wrong with my tests".
So, yes, it can be a cultural challenge. Other teams instantly see the benefits and use it enthusiastically.
Would you appreciate more collaborations with STAMP project partners? On which aspects?
While I would love to be involved in the STAMP project and its partners, mutation testing is currently something I can only work on in my spare time, which is very limited. That said, an area I currently find particularly interesting is mutant subsumption.
Biography
Henry Coles (@0hjc) is Head Of Development Practice at NCR Edinburgh, Scotland. He has been writing software professionally for 20 years, most of it in Java. Henry has produced many open source tools including pitest and an open source book, Java for Small Teams.