STAMP innovates running UI functional tests on various configurations and environments
How would you present STAMP?
STAMP is a European Research project with the aim of pushing the limits in Java software testing. Its novelty comes from the focus it has, which is to try to generate new tests based on existing tests. This is different from other initiatives trying to generate tests from source code only and this gives STAMP a much higher chance of getting tangible results.
STAMP develops three key technologies:
1) The usage of Mutation Testing as a way to measure existing test quality but more importantly as a strategy to mutate test code to generate new tests.
2) The ability to mutate Dockerfiles to execute test suites under various configurations and thus to automatically find out which configurations are supported by the software.
3) The ability to take a Java stack trace from production and to generate automatically a test that, when executed, leads to exactly the same stack trace! This means finding the conditions leading to the error and thus providing the developers help to fix the issue.
What is your role in STAMP?
XWiki contributes to STAMP mostly as a Use Case Provider, which means providing needs and use cases for the various tools being developed by STAMP. It also means testing the various tools developed by the Academic partners on a real production project with not only a sizable code base (hundreds of thousands of lines of code), but also a relatively well-tested code base (70% test coverage overall, more than 10K tests) and with a big focus on quality. Inside STAMP, XWiki also develop some tools and scripts such as:
- Jenkins pipeline script to handle flickering tests so that they are recognized as flickers and don’t generate false positives
- Maven plugin to compare OpenClover reports and that allows failing the build if the global test coverage contribution by a module is negative, across two dates.
- A functional test framework based on TestContainers, used in the XWiki code base to run XWiki’s Selenium tests across various configurations, inside Docker containers.
What key innovation do you bring or help to develop?
Key innovations:
- Automated UI testing using Docker and Selenium to reduce bugs related to configurations
- Increasing test quality through Mutation Testing. Code coverage doesn’t guarantee the quality of tests, just ensure that the code is reached but it doesn’t say anything about assertions nor whether the test is useful.
- New test generation (from production stack traces, from existing tests)
The main challenges we face are about generating new tests from existing ones and doing that in a reasonable timeframe. Another difficulty is in generating new tests that are relevant to the developers and discarding those that are not.
The most useful innovation for XWiki and the one we’re very excited about is the ability to run UI functional tests on various configurations/environments. We’ve already put this in production and we can run XWiki tests on Tomcat/Jetty/MySQL/PostgreSQL/HSQLDB/Chrome/Firefox/LibreOffice and all combinations of these environments and different versions of them. This has already allowed us to find several bugs that were only happening on some environments.
A word about yourself and your organization
I’ve been working for the XWiki open source project since 2005 and for the XWiki SAS company (as its CTO) since 2006. XWiki SAS is sponsoring the development of the XWiki open source project and I have the privilege to lead this team of talented developers. What’s interesting is that there’s a complete separation between the XWiki SAS company and the open source project, which has its own governance. We strongly believe in community-driven open source!
The XWiki open source project has always been very keen about software quality and over the years we’ve implemented a lot of testing strategies and tools to ensure this quality (check my blog post to know more about what we are working on). Thus, we were very happy to join the STAMP research project to try to push test quality even further and participate to the future of testing.