XWiki Leverages STAMP Software Testing Suite
What is it that characterises XWiki application lifecycle management?
XWiki is an open source project born 16 years ago, which is continuously improving through 40 new versions per year. This enterprise wiki provides a platform for application development and offers 700 extensions. All together, it contains more than one million lines of java and javascript code. Every month, nearly 50 contributors are working on XWiki (including translations), including around 10 developers regularly involved in the core of the platform. The Build phase is very tool-intensive. A lot of tests and verifications are carried out to ensure optimal quality and compatibility with previous versions.
Which of the automated testing tools in the STAMP project are used regularly by XWiki?
We perform integration tests, unit tests and functional tests on several browsers, with several versions of java, several DBMS and in various Servlet engines. The testing tools developed during the STAMP project help us to validate our test coverage, to improve quality. Each new code that joins an XWiki module is checked in the Build phase to ensure that the quality of the module will be equal or superior to that of the previous version.
CAMP is the flagship tool used in our integration chain. It supports the developers by amplifying their configuration tests in a Docker environment. We have a finite target list of environments to support, forming 16 configurations on which we run over 300 tests, for a total of 10 000 tests. When a user reports a problem with Tomcat and MySQL, for example, the developer no longer needs to install everything on his own computer. In five seconds, thanks to Docker and the continuous integration chain, he can launch the tests and identify the bug to correct it.
When a new feature is implemented, we can systematically check its behaviour with Oracle, MySQL or other DBMS. We have divided the configurations into three categories to keep the costs going down. The latest versions of all target environments are checked once a day. LTS (n-1) releases are run once a week and future releases once a month to provide early feedback on the problems to be solved and to plan the developer efforts. Thanks to Docker, these tests have helped to spread more experience of the infrastructure among the developers. This is important for us, as we are managing more and more infrastructure by ourselves.
Descartes works well. In our environment, we used this mutation testing tool regularly until May 2020 to ensure the quality of our test suite. Descartes found a few bugs, but we removed it from our continuous integration chain because we noticed stability concerns and high CPU resource consumption. False-positive errors required a lot of effort from the developers. We may use it again to improve it in the future. This is always a possibility. With it, developers can always check that the tests they write are of good quality.
What are the main benefits observed in use?
Automated tests allow the developer to test his own code before pushing it into the continuous integration chain. They help discovering bugs in all the target environments we are supporting. The whole Xwiki development team has become familiar with the STAMP tools, with everyone likely to write environment tests and ensure that all their tests pass. Our cloud team is also using Docker more frequently to deploy Xwiki on our own infrastructure.
The Holy Grail, in my opinion, would be to automate the whole release process. We're not quite there yet. The Xwiki team is looking to improve software engineering, through a myriad of automated testing experimentations. Some testing tools are proving to be very effective, others are giving more random results. Depending on the execution environment, they can reduce or increase the cost of fixing bugs. We've learned to check if the time spent using automated tests is relevant. This is a good practice for us now. And this is why we are ready to participate in other collaborative research projects focused on software quality.
A word about yourself and your organization
I’ve been working for the XWiki open source project since 2005 and for the XWiki SAS company (as its CTO) since 2006. XWiki SAS is sponsoring the development of the XWiki open source project and I have the privilege to lead this team of talented developers. What’s interesting is that there’s a complete separation between the XWiki SAS company and the open source project, which has its own governance. We strongly believe in community-driven open source!
The XWiki open source project has always been very keen about software quality and over the years we’ve implemented a lot of testing strategies and tools to ensure this quality (check my blog post to know more about what we are working on). Thus, we were very happy to join the STAMP research project to try to push test quality even further and participate to the future of testing.