What tests should go into nightly test suite?
Quite a few software engineering organizations split their sets of automatic tests into two or more suites. Decisions to start certain suites may be tied to engineers’ actions, such as a commit, a pull request, a merge, a release, a change of dependencies, etc. Collectively I will call these suites daily, for the reasons outlined below.
Other test suites may be tied to arbitrary timing events, such as nightly testing.
Let’s talk about the traditional decisions about which test cases go to which test suite, why they might not be the best criteria, and what alternatives exist.
TL;DR: migrate test cases that have long streaks of passed runs into nightly, regardless of the run lengths of those cases. Conversely, test cases that recently failed should be moved back from nightly to daily suites, regardless of their run length. This way, a subset of tests that alternate between pass and fail react to changes during the day, while tests that always pass do not slow you down during the day, yet are still run during the night, ready to be put back to daily.
What usually goes to the nightly test suite
Why is night such a magical time that it even has a test suite called after it? I believe the original idea was that, because not many changes arrive during the night (supposedly because humans sleep or are out of work), nighttime allows the computers (that run the tests) to be allocated for longer-latency tasks. As the next morning comes, results of nightly testing will be available for the returning humans to read through.
Because of that, a nightly test suite usually has these properties:
- It contains the largest number of tests. Oftentimes, it contains all test cases from daily suites, and then some more that are unique to the nightly.
- It contains very slow test cases. The motivation is that nobody wants to wait for those tests to finish during the daytime’s normal operations, but for some reason they are not deemed worthy of deletion, and so these slow tests are run when they won’t block humans.
To sum up the reasoning behind having a nightly test suite: “We have this ultimate test suite which we want to run. But it is long and we do not want to run it in a manner that could block human developers. So, we invented a compromise: run the ultimate test suite as infrequently as we can afford”.
Why it no longer makes sense to have a nightly
I believe the concept of a nightly test suite was invented a long time ago, way before continuous integration became the norm. Things were different in those days; the way we work and what we expect from our tests have changed. Let’s take a look.
For many software teams, there is no nighttime anymore
If you have a geographically distributed team, you would not be able to find a time interval when the whole team is outside the working hours. At least for some of your members, the “nightly” run becomes a test run that simply happens on a timer when they work. The magic of the night has disappeared.
The hope that nightly runs will not block human developers is futile
When a daily test suite is running, the associated code change it is testing is blocked from being merged until the test results come back.
Because all long tests have been exiled into the nightly, they seemingly won’t block anyone during the current day. But when the next morning comes and the nightly results come back red, the whole team pays the price of self-delusion: until all the nightly tests have been fixed, everyone should be blocked from merging new changes. Alternatively, it might be decided that the team simply ignores the results and continues operations as usual, and the task of “fixing the nightly” falls on the shoulders of the current poor sucker designated as the “build whisperer”, regardless of whose changes have caused the tests’ reaction.
Neither of these strategies — global blocking or devaluing results of the nightly tests — is good. This is especially bad compared to the daily suites: at most one developer is blocked by a failing personal daily suite result.
The daily test results are valued and cannot be ignored. The nightly test results are dreaded and swept under the rug if possible.
Computing resources are more elastic now
In the old times, most of the computing resources occupied during the day were free for running bigger test suites at night. Minimizing humans’ idle wait times (standing in a queue for a free computer) during the day was a priority simply because there were only so many hosts owned by the organization.
But the situation is often different now, and computing resources are much more elastic. What I mean is that it is possible now to buy compute time from a provider specifically for a test job, and release it immediately afterwards. The only exception to this is when your software under test needs very specific configurations such as special hardware or middleware.
There is little sense these days to use time-division multiplexing schemes such as night/day. All the tests can run all the time, if you are ready to deal with their results as soon as they are available.
The alternative: discriminate on test stability, not run length
The biggest issue with nightly suites is that long-running test cases end up there. But the length of a test case is irrelevant to its value to the release/merge conditions. Either a test case’s result is important, and a failing run must block further advancement until fixed, regardless of how long it takes to obtain the result; or the test case is not important, so it does not block anything, again meaning that its runtime is irrelevant.
We want feedback from our tests during the day, when we can see them. But not all test cases give the same amount of information when they are run. If we cannot run all the tests all the time, can we still select a subset that would give the most bang for the buck?
Is there any other criterion to prefer running some test cases, while delaying others, which is not based on their own run length? Yes, there is: it is their sensitivity to changes in the production code.
What should go to the nightly test suite
Remember why we run our tests: to look for reactions to changes made to the production code. The changes in production code are not uniform. Some subsystems may have stabilized and rarely change, while others are actively expanded, replaced, fixed, etc.
Properly focused test cases are mapped to those subsystems in the production code, and so their testing outcomes strongly correlate to the intensity of churn in those subsystems.
If nobody has touched a subsystem Foo, it would be weird to expect test cases closely associated with it to start reacting. They passed yesterday, and they will likely pass today. Running tests for Foo can likely wait until nightly.
But if another subsystem Bar is being actively modified at the moment, then we definitely want to be sure that those modifications do not break anything in its behavior. Test run history for Bar won’t predict future test runs for Bar, since it will be a different Bar. We do want to run tests associated with Bar frequently, without waiting for nightly.
Here are the conditions for defining which test cases go to the daily suite and which are moved to the nightly suite.
- If a test case from daily has a history of uninterrupted successful runs, it is moved to nightly.
- If a test case from daily has recently failed, it stays in daily.
- If a test case from nightly has started to fail, it is moved to daily.
With this approach, the test cases that give us most of the feedback are placed in daily to produce it early. Conversely, test cases that only say that “all is good” are moved to nightly. They are not completely disabled, however: as soon as a nightly test case breaks its no-reaction streak, it is returned to the daily suite.
How large should the threshold for moving a test case from daily to nightly be? How often is a nightly test allowed to fail without being moved back to daily? This is really a balancing game between the probability of blocking the whole team by a failed nightly versus paying the price of running the case for each developer during the day.
Nowhere in this strategy is the length of individual test cases taken into account.
Properties of the history-based nightly
The volume of the daily testing suite is proportional to the intensity of activity across the production code
Daily tests are selected based on recently modified production code, since they have recently reacted to changes in said production code. Most people in the team usually work on small subsets of production code, so they will only be interested in feedback from tests that actually exercise those subsets.
This is in fact a subcase of a more general idea. If your team has a mapping of test cases to production code, then, for every change in production code, only a subset of tests that exercise the modified code has to be run. All the remaining tests cannot possibly change their reaction to the production change, since they do not exercise the related code. But not every team has this test/production mapping at the ready. As a conservative approximation, all tests are run for every production change, which will give the same outcome as running that smaller subset would give, but takes longer, since many non-sensitive test cases are also invoked.
The dynamic daily/nightly separation uses the feedback from failing tests to discover the test/production mapping. Since we do not have the exact mapping, the nightly suite is used to construct an approximation of that mapping. The resulting daily subset is the result of such dynamic discovery.
Slow tests are dealt with properly. Having slower tests in daily runs is unpleasant for developers. That should motivate them to actually deal with those slow tests. Instead of shoving them out of sight into nightly (which does nothing to solve the problem, as I have shown), slow tests must be sped up by minimizing their dependencies, splitting them into faster sub-cases, etc.
Movement of tests follows changes in developers’ attention. A subsystem might stay unmodified for years, with its test cases happily returning success after success in nightly runs. But as soon as a business need arises to make changes to that subsystem, the subsystem and its tests again come back to focus. So, the associated test cases are soon moved back to the daily suite.
Of course, this dynamic rebalancing effect is only possible if your project has focused tests designed around subsystems found in the production code. If everything you have are end-to-end tests that depend on every production subsystem, then none of them will ever move to the nightly. End-to-end tests are simply too sensitive and fragile, for better or for worse.
Disarmed tests won’t hurt as much. A disarmed test case is one that cannot fail. It is always a signal of a bug hiding in a test: a forgotten early return, a botched assertion, etc. Since these cases cannot fail, they will eventually be migrated to nightly and will stay there forever. Surely, these buggy tests will be as useless there as they have been in daily before. At least they won’t contribute to the latency of daily testing.
Flaky tests will not be swept under the rug. Nondeterministic tests are the worst. Even a single one poisons test suites it is included in, undermining developers’ trust in those suites.
A flaky test cannot stay and hide in the nightly, blocking the whole team every morning. Instead, it will block individual developers in every daily run! Because it fails sporadically, it will be kept in the daily suite, where it will bother humans to no end, forcing them to deal with it.
But what to do with nondeterministic test cases? There is this idea of having a third special test suite where flaky tests end up. See the next section.
Are there any other test suites?
As soon as a test case has been identified as problematic, it is isolated in the quarantine suite, where other nondeterministic cases reside.
The idea of having a quarantine suite for flaky (nondeterministic) tests is described by Martin Fowler https://martinfowler.com/articles/nonDeterminism.html:
…it’s worthwhile to have a mechanism that ensures that tests don’t stay in quarantine too long. (…). One is a simple numeric limit: e.g. only allow 8 tests in quarantine. Once you hit the limit you must spend time to clear all the tests out. This has the advantage of batching up your test-cleaning if that’s how you like to do things. Another route is to put a time limit on how long a test may be in quarantine, such as no longer than a week.
Everybody hates flaky tests, but dealing with them is a pain. Moving them into nightly is the wrong move, since it does not solve the problem.
Isolation of the test cases means that their poisoning effect is contained.
The time-limited quarantine suite means that the team acknowledges that sometimes test cases will be deleted simply because the team does not have the resources to deal with them. The volume-limited quarantine suite means that the team has to make the hard decision and throw away some of the flaky tests before their accumulated instability negatively affects the project. Or better yet, to make another hard decision and allocate resources to fix whatever the reason for the nondeterminism is, e.g., buggy production code or overly sensitive tests.
A copy of the article on LinkedIn: https://www.linkedin.com/pulse/what-tests-should-go-nightly-test-suite-grigory-rechistov-aop1f