Resiliency Testing & Chaos Engineering
Deliberately injecting failure into a system to verify it actually behaves the way your architecture diagrams claim it will.
Want a visual for this topic?
Generate a diagram tailored to Resiliency Testing & Chaos Engineering — the AI picks whichever visual (architecture, flowchart, ER diagram, etc.) best fits this specific AWS concept.
Sign in to generate a visual →🎓 Learning objectives
- •Explain why an untested DR/resilience plan is a real risk, not just a formality
- •Explain what AWS Fault Injection Service does
- •Describe what a 'game day' exercise is and what it accomplishes
What is it?
Resiliency testing (often called chaos engineering) is the deliberate practice of injecting realistic failures — terminating an instance, blocking network access, throttling an API — into a system, in a controlled way, to verify it actually responds and recovers the way it's designed to, rather than assuming the architecture diagram's claims are true without ever having tested them.
Why it exists
A multi-AZ deployment, a configured DR plan, or an Auto Scaling Group's self-healing behavior are all claims about how a system will behave during a failure — but claims and configurations can be subtly wrong, incomplete, or have drifted since they were last verified, and the only way to know for certain is to actually cause the failure (safely, deliberately, in a controlled way) and observe what really happens. Resiliency testing exists because the difference between 'we believe this system is resilient' and 'we have verified this system is resilient' is enormous, and only becomes apparent — often catastrophically — during a real, unplanned incident if never tested beforehand.
Problem it solves
It solves the false-confidence problem (an untested DR plan or resilience configuration provides only the appearance of safety, not verified safety), and the incident-preparedness problem (a team that has practiced responding to a simulated failure is measurably faster and calmer during an actual incident than a team encountering that scenario for the very first time during a real outage).
Intuition
This is exactly like a fire drill: a building can have fire exits, alarms, and an evacuation plan on paper, but a fire drill is what actually verifies people know where to go, the doors actually open, and the alarm actually works — discovering a fire exit is blocked during an actual fire is far worse than discovering it during a planned drill.
Analogy
A hospital running a mock code blue (simulated cardiac emergency) drill: everyone knows the protocol on paper, but the drill verifies the crash cart is actually stocked, the team actually knows their roles under pressure, and the process actually works in practice — precisely so that when a real emergency happens, the team's response is practiced muscle memory rather than the first time anyone's actually done it.
Technical explanation
AWS Fault Injection Service (FIS) is a managed service specifically for running controlled chaos engineering experiments against AWS resources — predefined and custom actions can terminate EC2 instances, inject CPU/memory/disk stress, introduce network latency or packet loss, or throttle API calls, all with configurable stop conditions (automatically halting the experiment if a genuinely dangerous, unexpected impact is detected via a CloudWatch alarm, rather than letting an experiment run unchecked into an actual outage). A 'Game Day' is a structured, typically time-boxed exercise where a team deliberately simulates a specific failure scenario (sometimes using FIS, sometimes more manually) and practices their actual incident response — not just observing the technical system's behavior, but also testing whether the team's runbooks, communication, and escalation processes actually work under simulated pressure.
Architecture
A company runs quarterly Game Days simulating different failure scenarios: one quarter tests an Availability Zone failure (using FIS to terminate a significant fraction of instances in one AZ and observing whether Auto Scaling and load balancer health checks actually redistribute traffic within the expected time), another quarter tests a full Regional failover of their Pilot Light DR environment (actually executing their documented failover runbook, timing the real RTO achieved against their target, and identifying any gaps in the runbook's accuracy).
Workflow
- Start with low-risk, well-understood failure scenarios (terminating a single instance in a redundant fleet) before progressing to more significant tests (simulating an AZ or Region-level event). 2) Define clear stop conditions before running any experiment, so an unexpectedly severe impact automatically halts the test rather than escalating into a genuine incident. 3) Run tests during planned, communicated windows initially (not as a surprise to the whole organization), building confidence before potentially moving toward more advanced, less-announced testing. 4) Treat every finding — whether the system behaved as expected or revealed a gap — as valuable information, and actually fix identified gaps rather than just noting them. 5) Regularly repeat tests, since a system that was resilient six months ago may have silently drifted (a new dependency added without proper redundancy, a runbook that's gone stale) since it was last actually verified.
Example
A team confident in their RDS Multi-AZ setup runs an FIS experiment simulating an AZ failure during a planned test window, expecting the documented ~60-second failover. The actual test reveals failover took over 4 minutes — tracing the cause to an application connection pool configuration that wasn't retrying connections aggressively enough during the brief interruption, a gap that would have caused a much worse real-world outage than the 60-second target implied, discovered safely during a planned test rather than during an actual, unplanned AZ failure.
Real-world usage
Netflix's Chaos Monkey (part of their broader Simian Army tooling) pioneered much of the chaos engineering discipline industry-wide, deliberately and randomly terminating production instances specifically to force systems and teams to be genuinely resilient rather than only theoretically so; AWS Fault Injection Service was built to bring this same discipline to AWS customers as a managed, safer-to-operate service rather than requiring custom tooling.
Trade-offs
Running resiliency tests carries some inherent risk (a deliberately injected failure could, if misconfigured, cause more impact than intended) against the much larger risk of discovering a genuine gap for the first time during an actual, unplanned production incident — the controlled, planned nature of a well-run test (with stop conditions and a communicated window) is what makes this tradeoff favorable in practice. Starting with small, low-risk tests builds organizational confidence and tooling maturity before progressing to larger, more realistic (and higher-stakes) scenarios like a full Regional failover test.
Visual explanation
Picture a system architecture diagram with a small red 'X' deliberately injected at a specific point — perhaps terminating a specific EC2 instance in an Auto Scaling Group, or blocking network access from the application tier to the primary database. A resiliency test observes what actually happens next: does the Auto Scaling Group actually launch a replacement instance within the expected time? Does the application actually and gracefully handle the database being briefly unreachable, or does it crash outright? The test's value is in the honest answer to these questions, whatever that answer turns out to be.
Advantages
- —
Converts an assumed, untested resilience claim into a genuinely verified one, closing the gap between 'we think this works' and 'we know this works'
- —
Builds real incident-response muscle memory in teams before an actual, high-pressure incident occurs
- —
Often surfaces subtle, otherwise-hidden gaps (like the connection pool example) that pure architecture review or documentation reading would never catch
- —
AWS FIS's built-in stop conditions make running these experiments meaningfully safer than fully manual, ad-hoc chaos testing
Disadvantages
- —
A poorly planned or executed resiliency test can itself cause a genuine, unintended production incident if stop conditions and scope aren't carefully configured
- —
Requires real organizational buy-in and cultural comfort with deliberately breaking things in a controlled way, which not every organization is initially ready for
- —
Findings from resiliency tests require follow-through (actually fixing identified gaps) to provide real value — a test that surfaces a gap that's never addressed provides little benefit beyond the one-time insight
- —
Building meaningful, realistic test scenarios requires genuine understanding of the system's actual failure modes, not just running generic, superficial tests
Common mistakes
- —
Never testing a documented DR or resilience plan at all, discovering critical gaps only during an actual, high-stakes real incident
- —
Running a resiliency test without properly configured stop conditions, risking the test itself escalating into a genuine, unintended production outage
- —
Treating a resiliency test's findings as merely informational without actually following through to fix identified gaps
- —
Jumping straight to large-scale, high-risk tests (like a full Regional failover) without first building confidence and process maturity through smaller, lower-risk tests
- —
Running tests only once and assuming the system remains resilient indefinitely afterward, rather than recognizing that new changes and dependencies since the last test may have introduced new, unverified gaps
In the AWS Console
- 1
AWS Console → FIS (Fault Injection Service) → Experiment templates → Create experiment template
Select an action (e.g. 'aws:ec2:terminate-instances'), define the target resources (e.g. by tag, matching a specific fraction of an Auto Scaling Group), and configure a stop condition tied to a CloudWatch alarm.
Always configure a stop condition before running any experiment — this is what automatically halts the test if the impact is unexpectedly severe, rather than letting it run unchecked.
- 2
FIS → Experiment templates → [your template] → Start experiment
Run the experiment during a planned, communicated window, and closely monitor the target system's actual behavior and relevant CloudWatch metrics/alarms throughout.
Start with a small blast radius (e.g. a single instance or a small percentage of a fleet) before progressing to larger-scale tests as confidence builds.
🎤 Interview questions
Why is testing a disaster recovery or resilience plan important, beyond just having the plan documented and configured? (Listen for: configuration and documentation can be subtly wrong or drift over time; only actually testing verifies the real behavior matches the claimed/intended behavior.)
What does AWS Fault Injection Service do, and why does it include stop conditions? (Listen for: runs controlled chaos engineering experiments against AWS resources; stop conditions automatically halt an experiment if a CloudWatch alarm indicates unexpectedly severe impact, preventing the test itself from causing a genuine outage.)
What's a 'Game Day' exercise, and what does it test beyond just the technical system? (Listen for: a structured simulated-failure exercise; tests not just technical system behavior but also whether the team's actual incident response process, runbooks, and communication work under simulated pressure.)
How would you approach introducing resiliency testing to an organization that's never done it before? (Listen for: start with small, low-risk, well-understood scenarios; build confidence and process maturity before progressing to larger, higher-stakes tests like a full Regional failover.)
Describe a realistic scenario where a resiliency test revealed a gap that pure architecture review wouldn't have caught. (Listen for: a specific, plausible example — like an application-level configuration (e.g. connection pool retry behavior) that looks fine on paper but behaves unexpectedly under an actual injected failure.)