What is the testing pyramid?
Engineering culture and software practice
The testing pyramid is a rule of thumb for the shape of a test suite: many fast, cheap unit tests at the base, fewer integration tests in the middle, and a small number of slow, expensive end-to-end tests at the top. It was set out by Mike Cohn in Succeeding with Agile in 2009 and popularised by Martin Fowler. Get the shape wrong and you either test nothing that matters or spend all day waiting for brittle tests to finish.
What this means
Software can be tested at different levels. A unit test checks one small piece of code in isolation; it runs in a fraction of a second and, when it fails, points almost exactly at what broke. An integration test checks that several pieces work together. An end-to-end test drives the whole system the way a user would, clicking through a real interface; it is the most realistic and also the slowest, most fragile and hardest to diagnose.
The testing pyramid is a picture of how many of each you should have. A broad base of unit tests, a narrower band of integration tests, and a small cap of end-to-end tests. The shape is the message: most of your checking should happen at the cheap, fast, precise level, and only a little at the expensive, slow, realistic level.
It has a name because teams get the shape wrong in predictable ways, and the wrong shape has real consequences. A suite that is all end-to-end tests is slow and unreliable; a suite that is all shallow unit tests can pass while the product is plainly broken. The pyramid is a way for a non-engineer to reason about claims like we have tests or we have ninety per cent coverage without having to read a line of code.
Why it matters
When a team says we have tests, that sentence carries almost no information on its own. What matters is the shape and the economics behind it. A thousand fast unit tests that run in a minute give a team the confidence to change things many times a day. A few hundred end-to-end tests that take an hour and fail unpredictably give a team an excuse to stop running them, which is how a slow suite quietly stops being a safety net at all.
The pyramid matters to leaders because it translates a technical choice into money and speed. Tests at the base are cheap to write, quick to run and easy to diagnose, so they catch problems early when they are cheapest to fix. Tests at the top are expensive on every axis, so a suite that is top-heavy costs more, runs slower and breaks more often for reasons that have nothing to do with real faults. The shape is, in effect, a budget.
Coverage numbers deserve particular suspicion. A coverage percentage measures how much of the code a test suite touches, not how much it meaningfully checks. Once a coverage target becomes the goal, teams can hit the number with shallow tests that assert almost nothing, and the measure stops measuring what anyone cares about. That is the classic trap where a measure that becomes a target stops being a good measure.
The pyramid also maps neatly onto AI work. Judging an AI system, like testing software, is cheaper in bulk at the automated level and more expensive at the human level. A sensible evaluation strategy looks like a pyramid too: many cheap automated checks at the base, fewer expensive human reviews at the top. The shape reasoning transfers even when the thing being tested is a model rather than a program.
How it works
Where the term came from
The idea is credited to Mike Cohn, who described a test automation pyramid in his 2009 book Succeeding with Agile: Software Development Using Scrum, published by Addison-Wesley. Cohn set out three layers, with unit tests at the base, service or integration tests in the middle, and user-interface tests at the top, and argued that you should have far more of the low-level tests than the high-level ones because the lower tests are faster to run and cheaper to write. By his own and others' accounts the sketch predated the book; he had drawn it in conversation and described it at an agile gathering some years earlier, and a similar idea was arrived at independently by others around the same time.
The concept spread mainly through Martin Fowler, whose short bliki entry on the test pyramid, published in 2012, gave it the canonical name and made the case that high-level tests should be a second line of defence rather than the main event. A fuller practical treatment came in 2018 from Ham Vocke, in an article on Fowler's site, which worked through examples of each layer. It is worth noting that Cohn's original labels, unit, service, user interface, describe what is being tested and at what granularity, not simply how, a distinction later writers have laboured over.
The three layers and what each catches
Unit tests sit at the base because they are the foundation. Each one exercises a small piece of logic with everything else stubbed out, so it runs almost instantly and, when it fails, tells you almost exactly where the fault is. That precision is why the base is broad: fast, specific feedback is what lets a team work with confidence.
Integration tests occupy the middle. They check that units cooperate, that a service responds correctly to an input, that a component talks to a database as intended. They are slower and broader than unit tests, and when they fail the cause is less pinpointed, but they catch the class of problem that unit tests cannot see, namely the gaps between the pieces. Cohn himself called this the forgotten layer, because teams so often skip it.
End-to-end tests form the small cap. They drive the whole system as a user would and give the highest confidence that the thing actually works in the round. They are also the slowest to run, the most expensive to maintain and the most prone to failing for irrelevant reasons, which is why there should be few of them, reserved for the handful of journeys that matter most.
The economics of the shape
The pyramid is really an argument about cost and value per test. As you climb, each test buys more realism and less precision, takes longer to run, and breaks more often. A change deep in the pyramid gives quick, cheap, targeted signal; a change at the top gives slow, expensive, vague signal. A healthy suite spends most of its budget where the signal is cheapest and clearest, and rations the expensive realism at the top.
Software Engineering at Google, published by O'Reilly in 2020, reframes this in terms of resource footprint rather than labels. Google classifies tests as small, medium or large by what they are allowed to touch: a small test runs in a single process and thread with no network or disk, a medium test may use several processes on one machine, and a large test may run across machines. The characteristics being optimised are speed and determinism, and the book warns that as a suite approaches one per cent flakiness the tests begin to lose value; Google's own flaky rate hovers around 0.15 per cent, which still implies thousands of flakes every day. That ties the shape of a suite directly to whether it can be trusted.
Bad shapes and the arguments about them
The best-known failure is the ice cream cone, the pyramid turned upside down: a fat layer of slow end-to-end and manual tests on top and almost nothing underneath. Such suites are slow, unreliable and miserable to work with, and the pyramid exists largely to warn against this inversion. A related anti-pattern is the hourglass, plenty of unit and end-to-end tests but a starved middle, so that failures which a medium-scope test would have caught cheaply only surface in expensive end-to-end runs.
The pyramid has its critics, and they should be treated fairly. Kent Dodds proposed a testing trophy in 2018, developed from a much-quoted line by Guillermo Rauch, write tests, not too many, mostly integration, which adds static analysis at the bottom and puts more weight on integration tests, arguing that for modern front-end applications the integration level gives the best return on the time invested. Spotify's engineers popularised a honeycomb shape for microservices, again favouring the middle. These are practitioner proposals rather than settled findings, and they are best read as adjustments for particular contexts rather than replacements; the core warning, do not build an ice cream cone, survives all of them.
Coverage numbers and the AI-evaluation analogy
Coverage is the number leaders hear most and should trust least on its own. Ninety per cent coverage tells you the tests execute ninety per cent of the code; it says nothing about whether those tests would notice if the code were wrong. When a coverage figure becomes the target, teams can reach it with tests that assert little, so the number rises while real confidence does not.
The same shape reasoning applies to judging AI systems. Evaluating a model is cheap and fast when done by automated checks and expensive when done by human reviewers, exactly the trade-off the pyramid describes. A sensible AI evaluation strategy therefore looks pyramidal: a broad base of cheap automated checks run often, a small cap of costly human review reserved for the cases that matter most. The difference to keep in mind is that software tests, done well, are repeatable, whereas AI evaluations can themselves vary from run to run, so the base of that pyramid is less solid than it looks and needs its own reliability checks.
Examples
A retailer's small in-house team has spent two years automating tests entirely through the website, clicking through checkout in a browser. The suite takes forty minutes, fails a few times a day for reasons nobody can pin down, and is increasingly skipped before releases. An adviser recognises the ice cream cone and suggests pushing most of the checking down into fast unit and integration tests, keeping only three or four end-to-end journeys, such as a completed purchase and a declined card. The suite drops to a few minutes and starts being run again, which was the whole point.
A council department commissions a supplier to build a licensing portal and is told the work has ninety-five per cent test coverage. A cautious officer asks what shape the suite is and how much of that coverage is meaningful. It emerges that nearly all the tests are shallow unit tests that check trivial code and none exercise the integration with the payments provider, the one part most likely to fail in public. The officer asks the supplier to rebalance towards the middle of the pyramid rather than chase the coverage figure.
A charity is evaluating an AI assistant that drafts responses to enquiries. The team designs its evaluation like a pyramid: a large base of automated checks that flag obvious problems on every draft, a smaller band of structured spot-checks, and a small number of careful human reviews of sensitive cases. When someone proposes reviewing every draft by hand, the trustees point out that the expensive top of the pyramid should stay small, and that the cheap automated base is what makes the whole thing affordable.
Common misunderstandings
People treat the pyramid as a strict quota, as though a fixed ratio of tests were mandatory. It is a rule of thumb about proportion and shape, not a law with exact numbers. The point is that cheap, fast tests should dominate and expensive, slow ones should be rare, not that any particular percentage is sacred.
People assume more tests always means safer software. Volume is not the measure; shape and reliability are. A huge suite of slow, flaky end-to-end tests gives less real protection than a smaller, well-shaped suite that runs fast and can be trusted.
People read a high coverage number as proof of quality. Coverage measures how much code the tests run, not how much they meaningfully check. It is easy to hit a coverage target with tests that would not notice a real fault, which is why the number should never be taken on its own.
People confuse the pyramid with the tools used to build a suite. Unlike CI/CD, which is the pipeline that runs your tests, the testing pyramid is about the composition of the tests themselves. You can run a badly shaped suite through an excellent pipeline and still learn very little; the shape is a separate question from the plumbing.
People think the newer shapes, the trophy or the honeycomb, prove the pyramid is wrong. Rather than overturning it, these are context-specific adjustments that still reject the same failure, the top-heavy ice cream cone. The disagreement is about the middle, not about the warning.
Risks and boundaries
The pyramid is a heuristic, and it can be misapplied by taking it too literally. Its labels have been argued over for years: Cohn's unit, service and user interface describe granularity, and later writers dispute exactly where integration tests belong and what to call them. A team that spends its energy policing definitions rather than balancing speed against confidence has missed the point.
The shape is also not universal. Front-end-heavy applications, microservice architectures and data pipelines each stress different layers, which is why the trophy and honeycomb variants exist. Treating any one shape as gospel for every system is a mistake; the underlying economics, cheap and fast at the base, expensive and rare at the top, is more durable than any particular silhouette. Some practitioners argue the pyramid is an outdated economic model now that cheaper tooling has changed the cost of the middle layer, which is a live debate rather than a settled verdict.
Finally, the pyramid says nothing about whether the tests are any good. A well-shaped suite of weak tests, or a well-shaped suite riddled with flakiness, will still mislead. Shape is necessary but not sufficient; it has to sit alongside honest measurement of reliability and a sober reading of what coverage does and does not tell you.
What to do next
Ask what shape the suite is, not just whether tests exist. A simple question, roughly how many unit, integration and end-to-end tests do we have and how long do they take to run, will tell you more than any coverage figure. If almost everything is end-to-end, you are probably looking at an ice cream cone.
Ask how long the suite takes and how often it is actually run. A suite that takes so long it gets skipped before releases has stopped protecting anything, however impressive its size. Fast feedback at the base is what keeps a team running the tests at all.
Treat coverage as one input, never the target. When a team or supplier quotes a coverage percentage, ask what the tests would catch if the code were wrong, and resist turning the number into a goal, because a measure that becomes a target tends to stop being a good measure.
Match the strategy to the system rather than to a diagram. For a front-end-heavy product or a microservice estate, a heavier middle may be right; the question is whether the balance reflects where your risks actually are, not whether it matches a textbook triangle.
Apply the same shape thinking to AI evaluation. Build a broad base of cheap automated checks and reserve expensive human review for the cases that matter, and remember that the automated base can itself vary run to run, so it needs its own reliability check.
FAQs
What is the testing pyramid in one sentence?
A guideline that a test suite should have many fast unit tests at the base, fewer integration tests in the middle, and only a few slow end-to-end tests at the top.
Who came up with it?
Mike Cohn described the test automation pyramid in his 2009 book Succeeding with Agile, and Martin Fowler popularised it and fixed the name in a 2012 article.
What is the ice cream cone?
The pyramid inverted: a suite dominated by slow, brittle end-to-end and manual tests with too few unit tests. It is the main failure the pyramid exists to warn against.
Does ninety per cent coverage mean the software is well tested?
Not by itself. Coverage shows how much code the tests execute, not whether they would catch a real fault. A high number can sit on top of shallow tests that check very little.
How is the pyramid different from CI/CD?
The pyramid is about the mix of tests you have; CI/CD is the pipeline that runs them. A well-run pipeline executing a badly shaped suite still tells you very little.
Are the trophy and honeycomb replacements for the pyramid?
They are context-specific adjustments, mainly for front-end and microservice systems, that put more weight on integration tests. They still reject the top-heavy ice cream cone.
How does this relate to testing AI systems?
Judging an AI system is cheap in bulk with automated checks and expensive with human review, so a sensible evaluation strategy is pyramid-shaped, with many automated checks and few human reviews.
What is the single most useful question a leader can ask?
How long does the suite take, how often is it actually run, and what would it catch if the code were wrong. Those answers reveal more than any coverage percentage.