A production service went fully down when one cloud availability zone failed — diagnose the design flaw
A cloud availability zone had an outage this morning. Your service was completely down for its full duration, then recovered on its own when the zone came back. Leadership wants to know why a single zone took the whole service out. Here is the deployment.
Region: eu-1
Load balancer: single-zone (eu-1a only)
Web ASG: 6 instances, all in subnet eu-1a
Database: single instance, eu-1a, no standby
Cache: single node, eu-1a
Outage: eu-1a unavailable 09:12-10:40; service down the whole windowExplain why it went down and how you would redesign it.
Every tier lived in one zone, eu-1a, so when that zone failed the whole stack went down — six instances in one zone is redundancy against instance loss, not zone loss. Fix it by spreading the ASG, load balancer and a multi-AZ database with failover across two or three zones. Then losing a zone cuts capacity, not availability.
- ✗Blaming instance count when the flaw is single-zone placement
- ✗Claiming a region always fails as a whole, so only multi-region helps
- ✗Blaming the load balancer software instead of the single-AZ topology
- →What is the smallest change that would have kept the service up during this outage?
- →How would you prove the redesigned service survives a zone loss before it happens?
Solution
Root cause
Every tier — the load balancer, web ASG, database, cache — sits entirely in eu-1a. Losing one zone took it all down at once. Six instances are redundancy against losing an instance, not against losing a zone: they share one failure domain.
Redesign — treat the zone as a failure domain you survive
Load balancer: multi-AZ (eu-1a, eu-1b, eu-1c), health-checked targets
Web ASG: subnets in eu-1a/1b/1c, min per zone, auto-replaces nodes
Database: managed, multi-AZ, automatic failover to a standby
Cache: replica/cluster across zones (or degrade without it)Now losing eu-1a: the balancer drops its targets, the ASG relaunches capacity in eu-1b/1c, and the database fails over to its standby. The service cuts capacity, not availability.
Check
No second region is needed: a region's zones are independent failure domains with low-latency links between them, so multi-AZ is the baseline HA unit. A second region is for DR from a whole-region outage — costlier and more complex.
