What are flaky tests?
Engineering culture and software practice
Flaky tests are automated tests that pass and fail on the same code, with no change to the code they check, so a red result may mean nothing and a green one may hide a real fault. The term comes from industrial practice and was documented in academic work from 2014 and in Google's 2016 account of flakiness at scale. Re-running a test until it passes, the common reflex, is the anti-pattern that makes the problem permanent.
What this means
Automated tests exist to give a clear signal. Green means the change is safe to ship; red means something is broken and needs attention. A flaky test breaks that promise. It passes on one run and fails on the next, even though nobody touched the code it checks. The failure is not random in the deep sense; it has a cause, usually a timing gap or a shared resource, but it does not show up every time, so it looks like noise.
The pattern has a name because it is common, expensive and self-reinforcing. Once a team learns that a red build is often meaningless, people stop reading the red. They re-run the job, see green, and move on. The whole point of the test suite, to tell you the truth quickly, quietly erodes. A suite that lies half the time is worse than no suite at all, because it still consumes time and still carries the authority of a green tick.
Flaky tests are a property of the tests and the environment they run in, not of the bug being hunted. That is what makes them a distinct problem worth naming, and worth managing deliberately rather than tolerating.
Why it matters
For any organisation that builds, buys or depends on software, the test suite is the safety net that lets a team change things without breaking them. When that net becomes unreliable, two costs appear at once. The first is wasted time: engineers investigate failures that were never real, re-run builds, and wait for slow pipelines to settle. The second, and worse, cost is the failure that ships. When people are trained by experience to dismiss red as noise, they will eventually wave through a real fault, because it looked exactly like all the false alarms before it.
The scale of this is not trivial even at the most sophisticated firms. In its 2016 account, Google reported that about 1.5 per cent of all its test runs were flaky, that nearly 16 per cent of its tests showed some level of flakiness, and that 84 per cent of the pass-to-fail transitions its system flagged came from flaky tests rather than genuine breakage. If that is the experience of a company with world-class engineers, a smaller organisation with a hastily assembled suite should assume it has the same problem, only less well measured.
The topic has become sharper in the AI era. Tools that generate tests automatically can produce a large suite very quickly, and research has found that generated tests are at least as prone to flakiness as human-written ones. A big, fast-written suite that nobody trusts is not an asset; it is a liability that costs money to run and gives false comfort. The same pattern shows up in AI evaluations, the tests used to judge whether a model is behaving, which can return different verdicts on the same input from one run to the next.
For a leader, the practical question is simple: does your team trust its own tests, and can they show you the number that proves it? If the honest answer is that the build is often red and nobody worries, you are already paying for flakiness whether or not anyone has named it.
How it works
Where the term came from
Flaky tests come from industrial practice rather than a single coiner. The word was in working use among testers for years before it was studied properly. The first substantial academic treatment was An Empirical Analysis of Flaky Tests by Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, presented at the ACM Foundations of Software Engineering symposium in 2014, which examined real fixes and produced a widely cited taxonomy of root causes, with asynchronous waits, concurrency and test order dependency accounting for most cases.
The term reached a wider audience through Google. In May 2016 the engineer John Micco wrote a company blog post describing flakiness at Google's scale, reporting that about 1.5 per cent of test runs were flaky, that nearly 16 per cent of tests showed some flakiness, and that 84 per cent of the pass-to-fail transitions its continuous integration system caught were flaky rather than real. That post is a company source and should be read as an origin fact rather than an independent measurement, but its numbers have been corroborated repeatedly in later academic work, including A Survey of Flaky Tests, a comprehensive 2021 review by Owain Parry, Gregory Kapfhammer, Michael Hilton and Phil McMinn published in the journal ACM Transactions on Software Engineering and Methodology.
There is a family of older folklore about the bugs themselves that sits behind all this. A heisenbug changes when you look at it. A bohrbug is deterministic and solid, reproducible whenever the same conditions hold. A mandelbug has causes so tangled its behaviour looks chaotic. A schroedinbug appears to spring into existence only once someone reads the code and realises it should never have worked. These are cousins, catalogued in hacker jargon and in reliability research; flaky tests are the testing world's version of the same discomfort with non-determinism.
What actually makes a test flaky
Flakiness has recognisable causes, and naming them is the first step to fixing them. Timing and concurrency are the classic pair: a test starts an action and checks the result before the action has finished, so it passes when the machine is fast and fails when it is slow. Order dependence is another: one test quietly leaves state behind (a file, a record, a value in memory) and a later test passes or fails depending on whether it ran first. Shared state between tests running in parallel produces the same effect. External services, such as a network call to a third party, introduce failures that have nothing to do with your code. Resource limits, where a machine runs short of memory or file handles under load, cause intermittent collapses that look mysterious.
The common thread is that the triggering condition does not occur on every run. The bug is real and deterministic underneath; the appearance of randomness comes from a race, a dependency or an environment that varies. This is why marking a test as flaky and moving on is dangerous: the same instability that makes the test flake can be exactly the condition that will one day break the product.
How a flaky suite destroys trust
The damage is social as much as technical. A test suite works because people believe its verdict. The first time a red result turns out to be nonsense, an engineer forgives it. The tenth time, they stop looking. They develop the re-run-and-hope habit, and the suite quietly stops being read. Google's own account made the comparison to alarms that cry wolf; it is human nature to ignore a signal that has misled you before.
Once that happens, the suite has inverted its purpose. It still costs money to run, it still gates releases, and it still displays a reassuring green tick, but it no longer tells anyone the truth. The real fault, when it comes, arrives dressed as one more false alarm and sails through.
How experienced teams contain the damage
Mature teams treat flakiness as a managed problem, not a fact of life. The first move is to stop hiding it. The retry anti-pattern, automatically re-running a failed test until it passes, is understandable and disastrous in equal measure: it masks the instability and lengthens every build. Instead, disciplined teams quarantine a flaky test, moving it out of the gate that blocks releases and filing a ticket so the underlying cause is fixed, while accepting that quarantine itself can mask a genuine race condition if it becomes a dumping ground.
The second move is ownership and a fix-or-delete policy: every flaky test has a named owner, and a test that cannot be made reliable within an agreed window is either repaired or removed, because a test nobody trusts is worse than no test. The third move is measurement. Teams track a flake rate, the proportion of runs that produce an inconsistent result, so the problem becomes a number on a dashboard rather than an anecdote. A number can be argued about, budgeted for and driven down; an anecdote cannot.
The AI angle: generated tests and non-deterministic evaluations
AI-assisted development has made this more urgent. Test generation tools can write a very large suite very fast, and a 2024 study, Do Automatic Test Generation Tools Generate Flaky Tests? by Martin Gruber and colleagues, sampled 6,356 Java and Python projects and ran each test 200 times. It found that flakiness is at least as common in generated tests as in developer-written ones, although the suppression built into the EvoSuite tool cut the rate by 71.7 per cent. A leader should be sceptical of a sudden jump in test counts that is not matched by a demonstrated jump in trustworthy signal.
Separately, AI evaluations, the checks used to judge whether a model behaves, inherit a non-determinism of their own. Large language models can return different answers to the same prompt on different runs, so an evaluation that grades a model can itself flake. The lesson from the testing world transfers directly: measure the variability, do not paper over it with a re-run, and be honest about how much of your green is real.
Examples
A software team of eight at a logistics firm has a nightly build that is red more often than green. Nobody panics, because the failures are usually a handful of end-to-end tests that pass on a re-run. Over a quarter, a genuine regression in the pricing engine fails one of those same tests, an engineer assumes it is the usual noise, re-runs until green and ships. The overcharge reaches customers before anyone connects it to the ignored red. The fix, after the incident, is not more tests but a measured flake rate, a quarantine list and an owner for each unstable test.
A charity runs a donations platform maintained by a small agency. The agency proudly reports ninety per cent test coverage. When the trustees ask a plain question, how many of those tests give a reliable result, nobody can answer, because the flake rate has never been measured. The trustees ask for the number to be produced monthly. It turns out a meaningful slice of the suite is flaky, so the reassuring coverage figure was flattering the real state of the safety net.
A professional services firm adopts an AI coding assistant that can generate tests on request. Within weeks the suite triples in size and the build time doubles. Investigation shows many of the new tests fail intermittently because they were written against timing that the assistant could not know about. The firm keeps the generated tests that earn their place, deletes the ones that flake and cannot be stabilised, and adds a rule that any new batch of generated tests must clear a reliability check before it can gate a release.
Common misunderstandings
People assume a flaky test is a random failure with no cause. It is not. Every flaky test has a deterministic root cause, usually a race, a shared resource or a dependency; the failure only looks random because the triggering condition does not occur on every run. Treating it as bad luck is what lets it survive.
People think a flaky test is the same thing as the underlying bug. This is where the line with the heisenbug matters. A heisenbug is a fault in the software that changes its behaviour when you try to observe it. A flaky test is a property of the test and its environment, not of the bug being hunted; the code under test may be perfectly correct while the test still lies. This article is about the tests, unlike the heisenbug article, which is about the bugs.
People believe re-running until green is a reasonable coping strategy. It is the anti-pattern that makes flakiness permanent. Every re-run masks the instability, lengthens the build and trains people to dismiss red, which is precisely how a real fault eventually slips through.
People assume green coverage means the suite is trustworthy. Flaky tests count towards coverage even when nobody believes their result, so a high coverage figure can sit on top of a suite that produces unreliable signal. The number to ask for is the flake rate, not the coverage percentage.
People think AI-generated tests remove the problem because there are now so many of them. More tests, written faster, can mean more flakiness, not less. Volume without demonstrated reliability is a cost, not a safeguard.
Risks and boundaries
The term is sometimes stretched to cover any failure a team does not want to deal with. A test that fails because it was updated incorrectly is not flaky; it has a bug. A test that fails because the product genuinely broke is not flaky either. Flakiness is reserved for the specific case where neither the test nor the code under test changed and yet the result varies. Labelling every inconvenient red as flaky is how real regressions get buried.
There is a live debate about how far flakiness can ever be eliminated. Some experienced engineers argue that beyond a certain complexity, particularly in end-to-end tests of integrated systems, a residue of flakiness is inevitable, and that the same conditions that make such tests flaky are the conditions under which real bugs hide. On that view the goal is not zero flakiness but disciplined management: isolate it, measure it, and keep it away from the gates that block releases.
The folklore can also outrun the evidence in the other direction. Quarantine and retry are legitimate short-term containment, but treated as a permanent home they mask genuine faults. And the AI-era figures deserve care: claims that generated tests are flawless, or that a tool has abolished flakiness, are marketing rather than measurement. The honest position is that flakiness is a persistent property of large, fast-moving test suites and must be budgeted for, not wished away.
What to do next
Ask for the flake rate as a number, reported regularly. If a team says the suite is unreliable but cannot tell you what proportion of runs produce an inconsistent result, that is the first thing to fix. A measured rate turns a vague grumble into something you can budget against and drive down.
Ask what happens when the build goes red. If the honest answer is that people re-run until it passes, you have found the retry anti-pattern, and you should ask for a quarantine and fix-or-delete policy instead, with a named owner for each unstable test.
Treat coverage claims with polite scepticism. When a vendor or team quotes a coverage percentage, ask separately how much of that coverage produces a reliable signal. A high number sitting on a flaky suite is flattering, not reassuring.
Put a reliability gate in front of AI-generated tests. If your team is using a tool that writes tests, insist that a new batch must demonstrate stable results before it is allowed to block a release, so that speed of generation does not quietly buy you a bigger untrustworthy suite.
Extend the same thinking to AI evaluations. If you rely on automated checks to judge a model, ask how much the verdict varies from run to run, and make sure that variability is measured rather than hidden behind a re-run.
FAQs
What is the simplest definition of a flaky test?
A test that can both pass and fail on the same code, with no change to the code under test, so its result cannot be trusted on any single run.
Why not just re-run the test until it passes?
Because re-running masks the instability rather than fixing it, lengthens every build, and trains people to ignore red, which is exactly how a genuine failure eventually gets shipped.
Are flaky tests actually random?
No. Each has a deterministic cause, such as a timing gap, a shared resource or an external dependency. The failure only looks random because the triggering condition does not occur on every run.
How common are they?
Common enough to matter at every scale. Google reported that about 1.5 per cent of its runs were flaky, that nearly 16 per cent of tests were affected, and that 84 per cent of pass-to-fail transitions were flaky rather than real.
How is a flaky test different from a heisenbug?
A heisenbug is a fault in the software that changes when you observe it. A flaky test is a property of the test and its environment; the code being tested may be entirely correct.
Do AI-generated tests make this better or worse?
They can make it worse. A 2024 study found generated tests at least as flaky as human-written ones, so generating a large suite quickly can add cost and false comfort rather than reliable signal.
What single number should we track?
The flake rate, the proportion of test runs that produce an inconsistent result. It turns an anecdote into something you can manage and reduce.
What is a quarantine in this context?
Moving a known flaky test out of the gate that blocks releases, while a ticket is raised to fix its cause, so it stops disrupting everyone without simply being deleted and forgotten.