Test selection at Adyen: saving time and resources

If you work on a large-scale software system, odds are that a significant part of your build goes to test execution. It so often happens that you change one line of code but you get thousands of tests running, even though the majority of those tests don't even get close to that line of code.

This happens because build tools don't know much about what your tests really cover and because they are quite conservative: if you change a file in a module, it'll trigger all the tests of that module, all the tests of its dependencies, all the tests of the dependencies of your dependencies (your indirect dependencies), and so on and so forth.

Running all the tests we have available in our codebase for every code change is simply inefficient. Builds take longer to run, forcing developers to wait too long before their changes get merged (or before a test breaks). It also takes up too much of our CI infrastructure resources, which are large but finite. It's also less green as you are burning energy for literally zero business value. The situation gets even worse if one needs to start the whole platform to run the tests.

In practice, we know we don't have to run all the tests we have available in our codebase for every code change. The question then is: how to select which tests to run for a specific code change?

Test selection is a problem that software engineering researchers have been working on for decades now, but surprisingly there are no industry-wide solutions yet; therefore, companies might need to implement their own services for that. In this blog post, we'll:

• Explain in a nutshell how our test selection tooling works.

• Dive into the technical implementation details of our solution.

• Show the benefits in terms of test case reduction and pipeline speed up.

• How we keep the tool up and running reliably

• The lessons we learned while building the tool.