What is chaos engineering?
Engineering culture and software practice
Chaos engineering is the discipline of deliberately injecting failures into a system, ideally in production and under control, to find out whether it truly tolerates them before customers do. Pioneered at Netflix around 2010 with a tool called Chaos Monkey and codified in the Principles of Chaos Engineering, it treats resilience as something to be tested experimentally rather than assumed. A game day is the scheduled, team-wide version, tracing back to Amazon's GameDay exercises in the mid-2000s.
What this means
Most organisations assume their backups work, their failovers kick in and their systems cope when something breaks, right up until the day something breaks and they find out otherwise, usually at the worst possible moment. Chaos engineering flips that around: instead of waiting for the real failure, you cause a controlled one on purpose, while you are watching and ready, and see what actually happens.
It is best understood as an experiment, not vandalism. You start by defining what normal looks like, predict that normal will hold when you inject a particular failure, then cause that failure in a limited way and check whether your prediction was right. If the system copes, you have earned confidence. If it does not, you have found a weakness on a quiet day of your choosing rather than during a real crisis.
The scheduled, human version is the game day: a planned exercise where a team rehearses its response to a simulated disaster. It tests not just the technology but the people and the procedures, which is often where the real gaps are.
Why it matters
The practical case is simple: it is far cheaper to discover that your payment provider failover does not work on a quiet Tuesday than on the busiest trading day of the year. Chaos engineering surfaces the hidden assumptions and untested dependencies that cause real outages, and it does so under conditions you control, with the people who can fix things already paying attention.
For most small organisations the direct value is not running failure experiments in live systems, which requires considerable maturity, but the mindset behind it. The first chaos experiment a small firm should run is restoring from backup: many organisations discover only in a real emergency that their backups were incomplete or unrecoverable. A supplier-failure drill, asking what the team would actually do if this critical service went dark, is the same idea applied on paper.
AI dependencies make this newly relevant. Organisations are wiring model APIs into their processes, and those services can go down, slow to a crawl or degrade in quality. A business that depends on an AI service without ever testing what happens when it is unavailable is making exactly the untested assumption chaos engineering exists to expose. A simple drill, turning off the dependency and seeing whether the process has a sensible fallback, is worth more than any assurance.
In regulated sectors this has moved from good practice to expectation. UK financial regulators now require firms to identify important business services, set tolerances for disruption and test their ability to stay within them under severe but plausible scenarios. That is chaos engineering's logic written into supervisory rules, and while it applies formally to financial firms, the underlying discipline of testing resilience by simulating failure is spreading well beyond them.
How it works
Where the term came from
Chaos engineering grew out of Netflix's move to cloud infrastructure around 2010, when the company built Chaos Monkey, a tool that randomly disabled production servers to force engineers to build systems that could survive such failures. It was open-sourced in 2012 and joined by a wider "Simian Army" of failure-injection tools. The discipline was named and formalised as the practice matured: the Principles of Chaos Engineering statement defines it as experimenting on a system to build confidence in its capability to withstand turbulent conditions in production, and a peer-reviewed article, "Chaos Engineering" by Ali Basiri and colleagues in IEEE Software in 2016, set out the approach for a technical audience. The related practice of game days is older, tracing to Amazon's GameDay programme created in the early 2000s by Jesse Robbins, whose title was Master of Disaster and who drew on his training as a firefighter; the 2012 ACM Queue conversation "Resilience Engineering: Learning to Embrace Failure" records that history. The intellectual lineage runs back to resilience engineering as studied by Erik Hollnagel and David Woods. These companies are named here as origin facts only.
The principles
The Principles of Chaos Engineering set out an ideal method. Define the steady state as a measurable output that indicates normal behaviour. Hypothesise that this steady state will continue in both a control group and an experimental group. Introduce variables reflecting real-world events, such as servers crashing, connections dropping or a dependency failing. Try to disprove the hypothesis by looking for a difference. The harder it is to disrupt the steady state, the more confidence the system has earned. Two further principles matter: automate experiments so they run continuously, and, crucially, minimise the blast radius so an experiment harms as few users as possible while still being informative.
What a game day looks like
A game day is a scheduled exercise where a team deliberately triggers or simulates a failure and works through the response together. It might take down a component in a test environment, or simply talk through a scenario as a tabletop exercise. The value is in exposing the gaps that only appear under pressure: the runbook that is out of date, the login nobody has, the person who is on holiday, the dependency no one realised was critical.
Prerequisites and how experienced teams handle it
Chaos experiments in production are not a starting point. They require observability, so you can see what the experiment does; the ability to roll back or stop quickly; and blast-radius control, so a surprise cannot cascade. Mature teams start small, a single component or a tiny fraction of traffic, with automatic stop conditions, and widen only as confidence grows. For most small firms the sensible entry points are testing backup restores and running dependency drills, not injecting faults into live systems.
Examples
A retailer worries about what would happen if its payment provider failed during a seasonal peak. Rather than hope, it runs a game day in a test environment: it simulates the provider being unreachable and watches whether checkout falls back gracefully to a queue or simply collapses. It discovers the fallback was never finished, fixes it in the quiet season, and enters its busiest period with tested confidence rather than a hopeful assumption.
A charity relies on nightly backups of its case-management data but has never restored one. As its first chaos experiment, it schedules a restore drill: it takes a recent backup and attempts a full recovery onto a spare system. The drill reveals that one crucial dataset was never included in the backup job. Finding this in a planned exercise, rather than after a ransomware incident, is precisely the point of the practice.
A software team of eight has built an internal tool around an external AI model API. During a game day the team deliberately blocks the API and watches the tool. It finds that instead of degrading gracefully, the tool hangs and blocks other work. The team adds a timeout and a clear fallback message, so that when the real service degrades weeks later, the tool stays usable and staff know what has happened.
Common misunderstandings
The first misconception is that chaos engineering means randomly breaking things in production. It does not; it is a controlled experiment with a hypothesis, a limited blast radius and the ability to stop. Breaking things without a hypothesis or a safety net is not chaos engineering, it is an incident you caused.
The second is that it is only for giant technology companies. The mindset scales down: a backup restore test and a supplier-failure drill are chaos experiments a two-person organisation can run, and they deliver most of the value for a fraction of the sophistication.
The third is that it replaces other testing. It does not; it complements it. Traditional testing checks whether code does what it should under expected conditions, whereas chaos engineering checks how the whole system behaves under unexpected failure. This is also what separates it from red teaming: red teaming is adversarial testing, a simulated attacker probing security or an AI system for weaknesses, whereas chaos engineering injects non-adversarial failures such as crashes and outages to test resilience. Both deliberately stress a system, but one models an attacker and the other models bad luck.
The fourth is that the goal is to cause maximum disruption. The opposite is true: the discipline explicitly requires minimising blast radius, causing the smallest disruption that still yields a useful finding. Bravado is an anti-pattern.
The fifth is that you should surprise your own team to make the test realistic. Surprising the people who must respond, or breaking something you cannot fix, are recognised anti-patterns that damage trust and can turn an experiment into a genuine crisis.
Risks and boundaries
The central risk is running experiments before you are ready. Without observability you cannot tell what your experiment did; without rollback you cannot stop it; without blast-radius control a small test can cascade into a real outage. The Principles are explicit that a poorly designed experiment can cause exactly the production failure it was meant to prevent, and the practitioner's responsibility is to understand and contain that risk. A small firm that injects faults into live systems before mastering restores and drills has the order wrong.
A second boundary is that chaos engineering tests resilience, not correctness or security in the adversarial sense. It will tell you whether your system copes when a component dies; it will not tell you whether your code is right or whether an attacker could break in, which are the jobs of other kinds of testing. Treating it as a substitute for those leaves real gaps.
The regulatory dimension carries its own caveat. UK operational-resilience rules require testing against severe but plausible scenarios, and while this validates the discipline, meeting a regulatory expectation is a matter for each firm and its own advisers; running a game day does not by itself demonstrate compliance with any particular rule. The honest framing is that chaos engineering supports operational resilience, not that it guarantees a regulatory standing.
What to do next
Start with restores, not production experiments. Schedule a test that actually recovers your most important data from backup onto a separate system. If it works, you have earned real confidence; if it does not, you have found the most important gap you have, on your own terms.
Run a dependency drill for each critical supplier and tool, including any AI service. Ask, and where safe test, what actually happens if this goes dark or degrades, and make sure there is a sensible fallback and a clear message for the people affected.
Hold a tabletop game day for a plausible disaster. Gather the people who would respond, walk through the scenario, and note every gap: the out-of-date runbook, the missing login, the unclear decision rights. Fix the gaps and repeat periodically, because they reappear as systems change.
Build the prerequisites before attempting anything in live systems: know how to observe what is happening, know how to stop and roll back, and contain the blast radius. Never surprise your own responders and never break something you cannot restore.
FAQs
What is chaos engineering in simple terms?
Deliberately causing a controlled failure in a system to see whether it copes, so you find weaknesses on your own schedule rather than during a real crisis.
Where did it come from?
From Netflix around 2010, with a tool called Chaos Monkey that randomly disabled servers. It was formalised in the Principles of Chaos Engineering and a 2016 IEEE Software article.
What is a game day?
A scheduled, team-wide exercise rehearsing the response to a simulated failure. The idea traces to Amazon's GameDay programme in the early 2000s.
Is this only for big technology firms?
No. The mindset scales down. A backup restore test and a supplier-failure drill are chaos experiments any small organisation can run.
What should a small business try first?
A backup restore drill and a dependency drill. Confirm you can actually recover your data and that critical suppliers, including AI services, have a tested fallback.
How is it different from red teaming?
Chaos engineering injects non-adversarial failures like crashes and outages to test resilience. Red teaming simulates an attacker probing for security weaknesses.
Isn't deliberately breaking things dangerous?
It can be, which is why the discipline requires a hypothesis, observability, the ability to stop, and a minimised blast radius. Breaking things without those is not chaos engineering.
Does it help with regulatory operational resilience?
It supports the testing that resilience rules expect, but running a game day does not by itself demonstrate compliance with any particular regulation.