Infrastructure System Design
Senior scenarios: HA stateless services, zero-downtime deploys and schema migrations, centralized logging, capacity planning, multi-region, secrets and rate limiting.
9 questions
SeniorDesignVery commonA three-tier web application — a web/UI tier, a stateless application tier, and a single relational database — runs one server per tier today, so any single failure takes the whole product offline. Redesign it for high availability with no single point of failure. Cover how you make the application tier horizontally redundant and genuinely stateless (where session and uploaded-file state go), how traffic reaches only healthy instances, how you remove the database as a single point of failure (a primary with replicas and automated failover), and how the system degrades gracefully when one tier is partly down. Name the single points of failure you remove at each tier and any you knowingly accept.
A three-tier web application — a web/UI tier, a stateless application tier, and a single relational database — runs one server per tier today, so any single failure takes the whole product offline. Redesign it for high availability with no single point of failure. Cover how you make the application tier horizontally redundant and genuinely stateless (where session and uploaded-file state go), how traffic reaches only healthy instances, how you remove the database as a single point of failure (a primary with replicas and automated failover), and how the system degrades gracefully when one tier is partly down. Name the single points of failure you remove at each tier and any you knowingly accept.
Run every tier redundantly: several stateless app instances behind a load balancer that health-checks targets and routes only to healthy ones, with session and uploaded-file state moved to a shared store so any instance serves any request. Make the database a primary with replicas and automated failover, offloading reads to replicas. Add graceful degradation so a failing tier sheds load instead of cascading.
Common mistakes
- ✗Treating a higher instance count as high availability while a single DB remains
- ✗Pinning session state in instance memory so instances are not interchangeable
- ✗Making the web tier redundant but leaving the database a single point of failure
Follow-up questions
- →How does moving session state to a shared store make instances interchangeable?
- →What breaks if the database primary fails and there is no automated failover?
SeniorDesignCommonA read-heavy web application shows the same content to many users but hits its database on nearly every request, and read latency is climbing. Design a layered caching strategy. Cover a CDN or edge cache for static and cacheable responses (driven by HTTP cache headers), an application-layer cache (e.g. Redis) for hot computed data and query results, and read replicas for the database. The hard part is invalidation and consistency: how you set TTLs, how you invalidate or update entries when the underlying data changes, and how you avoid a cache stampede when a hot key expires. Explain the trade-off between serving slightly stale data cheaply and the complexity of keeping every cache layer correct.
A read-heavy web application shows the same content to many users but hits its database on nearly every request, and read latency is climbing. Design a layered caching strategy. Cover a CDN or edge cache for static and cacheable responses (driven by HTTP cache headers), an application-layer cache (e.g. Redis) for hot computed data and query results, and read replicas for the database. The hard part is invalidation and consistency: how you set TTLs, how you invalidate or update entries when the underlying data changes, and how you avoid a cache stampede when a hot key expires. Explain the trade-off between serving slightly stale data cheaply and the complexity of keeping every cache layer correct.
Cache in layers: a CDN/edge cache for static and cacheable responses driven by HTTP cache headers, an application cache like Redis for hot data and query results, and read replicas. Set TTLs by each data's staleness tolerance, invalidate or update entries when the source changes, and guard hot keys against a stampede on expiry. The trade-off is serving slightly stale data cheaply versus the complexity of keeping every layer correct.
Common mistakes
- ✗Caching with no invalidation so entries stay stale after the source changes
- ✗Letting a hot key expire and stampede the database with a rebuild herd
- ✗Treating CDN, app cache, and replicas as one layer needing no separate reasoning
Follow-up questions
- →How do you invalidate a cached entry the moment its underlying data changes?
- →What is a cache stampede, and how do you stop a hot key's expiry from causing one?
SeniorDesignCommonAbout 500 hosts each emit application and system logs, and engineers currently SSH into individual machines to read them. Design a centralized logging pipeline. Cover collection (an agent on every host), transport with buffering so a downstream slowdown neither drops logs nor blocks the applications, the store and its index for fast queries, and — because this is where logging bankrupts a budget — retention tiers and volume/cardinality control (what you keep hot vs archive vs drop). Explain the trade-off between indexing everything for instant search and the storage cost that implies, and how you would roll the agent out to 500 hosts consistently. Note why unbounded high-cardinality fields make an index both slow and expensive to keep.
About 500 hosts each emit application and system logs, and engineers currently SSH into individual machines to read them. Design a centralized logging pipeline. Cover collection (an agent on every host), transport with buffering so a downstream slowdown neither drops logs nor blocks the applications, the store and its index for fast queries, and — because this is where logging bankrupts a budget — retention tiers and volume/cardinality control (what you keep hot vs archive vs drop). Explain the trade-off between indexing everything for instant search and the storage cost that implies, and how you would roll the agent out to 500 hosts consistently. Note why unbounded high-cardinality fields make an index both slow and expensive to keep.
Put a collection agent on every host shipping to a buffered transport (a queue or broker) so a downstream slowdown neither drops logs nor blocks the app. Land them in an indexed store, but tier retention — hot and indexed for recent data, cheap archive for older, drop noise — since indexing everything bankrupts the budget. Bound high-cardinality fields, and roll the agent out with config management so all 500 hosts match.
Common mistakes
- ✗Writing logs synchronously to the store so a slow store blocks the application
- ✗Indexing and keeping everything hot forever instead of tiering retention
- ✗Adding unbounded high-cardinality fields that bloat and slow the index
Follow-up questions
- →Why put a buffer between the agents and the store rather than writing directly?
- →How do retention tiers cut cost without losing the ability to investigate old incidents?
SeniorDesignCommonLeadership asks for a disaster-recovery plan for a stateful service after a near-miss where a bad migration corrupted production data. Design the DR plan. Define RPO and RTO and what business tolerance sets each, then choose mechanisms that meet them: backup frequency and type driving RPO, restore and failover speed driving RTO. Cover how backups are taken and — critically — how restores are regularly tested (an untested backup is not a backup), where backups live so one failure cannot destroy both the data and its backups, and a written, rehearsed failover runbook. Explain the difference between RPO (how much data you can afford to lose) and RTO (how long you can afford to be down), and why they drive different investments.
Leadership asks for a disaster-recovery plan for a stateful service after a near-miss where a bad migration corrupted production data. Design the DR plan. Define RPO and RTO and what business tolerance sets each, then choose mechanisms that meet them: backup frequency and type driving RPO, restore and failover speed driving RTO. Cover how backups are taken and — critically — how restores are regularly tested (an untested backup is not a backup), where backups live so one failure cannot destroy both the data and its backups, and a written, rehearsed failover runbook. Explain the difference between RPO (how much data you can afford to lose) and RTO (how long you can afford to be down), and why they drive different investments.
Set RPO (tolerable data loss) and RTO (tolerable downtime) from business tolerance, then pick mechanisms for them: backup frequency and type drive RPO; restore and failover speed drive RTO. Take backups on that cadence, store them isolated from the primary so one failure cannot destroy both, and — critically — restore-test them regularly, since an untested backup is not a backup. Keep a rehearsed runbook so recovery is not improvised.
Common mistakes
- ✗Conflating RPO (data loss) and RTO (downtime) as one number
- ✗Storing backups beside the primary so one failure destroys both
- ✗Never restore-testing backups, so recovery fails when it is actually needed
Follow-up questions
- →How do backup frequency and restore speed map onto RPO versus RTO?
- →Why is a backup you have never restored not really a backup?
SeniorDesignCommonYou are the platform team for a microservices system of about 40 services, and debugging an incident today means SSHing into boxes to grep logs. Design an observability stack the whole platform will share. Cover the three pillars — metrics, logs, and traces — what each is best at and what each costs; how you instrument services consistently (a shared library or sidecar) so one request can be followed across service hops; which signals you alert on (symptom-based — SLO burn and the RED/USE signals) versus which you keep only for debugging; and how you keep metric cardinality from exploding your time-series database. Explain why alerting on every low-level cause instead of user-facing symptoms creates pager fatigue and slows real incidents.
You are the platform team for a microservices system of about 40 services, and debugging an incident today means SSHing into boxes to grep logs. Design an observability stack the whole platform will share. Cover the three pillars — metrics, logs, and traces — what each is best at and what each costs; how you instrument services consistently (a shared library or sidecar) so one request can be followed across service hops; which signals you alert on (symptom-based — SLO burn and the RED/USE signals) versus which you keep only for debugging; and how you keep metric cardinality from exploding your time-series database. Explain why alerting on every low-level cause instead of user-facing symptoms creates pager fatigue and slows real incidents.
Use all three pillars for their strengths: metrics for cheap aggregate trends and alerting, logs for per-event context, traces for following a request across services. Instrument every service identically with propagated trace IDs. Alert on user-facing symptoms — SLO burn and the RED/USE signals — not every cause, and keep the rest for debugging. Cap label cardinality so the time-series database does not explode.
Common mistakes
- ✗Treating metrics, logs, and traces as interchangeable rather than complementary
- ✗Alerting on low-level causes instead of user-facing symptoms, causing pager fatigue
- ✗Ignoring metric cardinality until the time-series database falls over
Follow-up questions
- →Why alert on SLO burn rather than on a single service's CPU spike?
- →How does a high-cardinality label like user ID break a time-series database?
SeniorDesignCommonAcross a fleet of services, database passwords and API keys are currently pasted into environment files and committed config, so a single leaked repo or host exposes everything. Design a fleet-wide secrets-management architecture. Cover a central secret store as the single source of truth, how services authenticate to it and fetch secrets at runtime instead of baking them into images, how you rotate credentials with little or no downtime, how you limit blast radius so one compromised service cannot read every secret (scoped, least-privilege access), and how you keep plaintext secrets out of code, logs, and CI. Explain why short-lived, dynamically issued credentials shrink the damage of a leak compared with long-lived shared ones.
Across a fleet of services, database passwords and API keys are currently pasted into environment files and committed config, so a single leaked repo or host exposes everything. Design a fleet-wide secrets-management architecture. Cover a central secret store as the single source of truth, how services authenticate to it and fetch secrets at runtime instead of baking them into images, how you rotate credentials with little or no downtime, how you limit blast radius so one compromised service cannot read every secret (scoped, least-privilege access), and how you keep plaintext secrets out of code, logs, and CI. Explain why short-lived, dynamically issued credentials shrink the damage of a leak compared with long-lived shared ones.
Keep every secret in a central store as the single source of truth; each service authenticates as itself and fetches secrets at runtime, never baked into images. Scope access per service by least privilege so one compromise cannot read everything, and prefer short-lived, dynamically issued credentials that auto-expire, shrinking blast radius and making rotation routine, not a redeploy. Keep plaintext out of code, logs, and CI.
Common mistakes
- ✗Committing plaintext secrets and relying on repo encryption to hide them
- ✗Handing every service one shared credential instead of scoped, least-privilege access
- ✗Using long-lived static secrets and treating rotation as a rare emergency
Follow-up questions
- →Why does a short-lived credential limit the damage of a leaked secret?
- →How does per-service scoping bound the blast radius of one compromised service?
SeniorDesignOccasionalA consumer service sees unpredictable, spiky traffic — quiet for hours, then a 10x surge in seconds when a post goes viral — and you must design an autoscaling strategy that balances cost against enough headroom to absorb spikes. Cover which signal you scale on (CPU vs a request or queue-depth metric closer to real load), why purely reactive autoscaling can lag an instant spike (instance boot and warm-up time) and how you compensate (a warm buffer, pre-scaling on schedules or predictions), how you set min/max bounds and scale-in protection to avoid flapping, and the cost-vs-headroom trade-off itself. Explain why running near 100% utilization to save money leaves no room to absorb the very spikes this service is defined by.
A consumer service sees unpredictable, spiky traffic — quiet for hours, then a 10x surge in seconds when a post goes viral — and you must design an autoscaling strategy that balances cost against enough headroom to absorb spikes. Cover which signal you scale on (CPU vs a request or queue-depth metric closer to real load), why purely reactive autoscaling can lag an instant spike (instance boot and warm-up time) and how you compensate (a warm buffer, pre-scaling on schedules or predictions), how you set min/max bounds and scale-in protection to avoid flapping, and the cost-vs-headroom trade-off itself. Explain why running near 100% utilization to save money leaves no room to absorb the very spikes this service is defined by.
Scale on a signal close to real load — requests or queue depth, not just CPU. Reactive autoscaling lags an instant spike by boot and warm-up time, so keep a warm headroom buffer and pre-scale on schedules or predictions. Set min/max bounds and scale-in protection so it does not flap. The trade-off is cost vs headroom: near-100% utilization saves money but leaves nothing to absorb the defining spikes.
Common mistakes
- ✗Assuming reactive autoscaling reacts instantly, ignoring boot and warm-up lag
- ✗Running at 100% utilization with no headroom for the defining spikes
- ✗Scaling only on CPU when requests or queue depth track real load better
Follow-up questions
- →Why can reactive autoscaling alone fail to save you during an instant 10x spike?
- →How does a warm buffer or scheduled pre-scaling change your cost-vs-headroom balance?
SeniorDesignOccasionalBehind your API gateway sits a fleet of stateless gateway instances, and you must enforce a global per-client rate limit — say 1000 requests per minute per API key — that holds no matter which instance a request lands on. Design the rate limiter. Cover the algorithm (e.g. token bucket vs a fixed/sliding window) and what it allows for bursts, why a per-instance in-memory counter cannot enforce a global limit, how instances share state through a fast central store using atomic operations, how you keep the limiter's own latency and availability from hurting the request path, and what you do when the shared store is unreachable (fail-open vs fail-closed). Explain the trade-off between strict global accuracy and the extra latency of a shared-counter round trip on every request.
Behind your API gateway sits a fleet of stateless gateway instances, and you must enforce a global per-client rate limit — say 1000 requests per minute per API key — that holds no matter which instance a request lands on. Design the rate limiter. Cover the algorithm (e.g. token bucket vs a fixed/sliding window) and what it allows for bursts, why a per-instance in-memory counter cannot enforce a global limit, how instances share state through a fast central store using atomic operations, how you keep the limiter's own latency and availability from hurting the request path, and what you do when the shared store is unreachable (fail-open vs fail-closed). Explain the trade-off between strict global accuracy and the extra latency of a shared-counter round trip on every request.
Keep limit counters in a fast shared store like Redis that all stateless instances update atomically, since a per-instance in-memory counter only limits that one instance, not the global key. Enforce the average rate while allowing controlled bursts (a token bucket). Bound the check's latency, and when the store is unreachable choose fail-open or fail-closed deliberately. Stricter global accuracy costs a shared round trip per request.
Common mistakes
- ✗Using per-instance counters that let a client get N times the limit across N instances
- ✗Doing read-modify-write on the shared counter without atomic operations
- ✗Leaving fail-open vs fail-closed undecided when the shared store is unreachable
Follow-up questions
- →Why does a per-instance counter let a client exceed the intended global limit?
- →When would you choose fail-open over fail-closed if the counter store goes down?
SeniorDesignOccasionalA fleet of application servers behind a load balancer must be deployed to a new version that also changes the relational database schema — renaming and splitting a column — with zero downtime and a safe rollback. Design the release. Cover how you sequence the schema change against the code rollout so old and new app versions run simultaneously against one database (the expand/contract, backward-compatible-migration pattern), how the rolling deploy drains and replaces instances without dropping requests, what keeps each migration step forward- and backward-compatible, and how you roll back if the new version misbehaves after some data has already been written. Explain why a single destructive ALTER shipped in one release with the code would cause an outage.
A fleet of application servers behind a load balancer must be deployed to a new version that also changes the relational database schema — renaming and splitting a column — with zero downtime and a safe rollback. Design the release. Cover how you sequence the schema change against the code rollout so old and new app versions run simultaneously against one database (the expand/contract, backward-compatible-migration pattern), how the rolling deploy drains and replaces instances without dropping requests, what keeps each migration step forward- and backward-compatible, and how you roll back if the new version misbehaves after some data has already been written. Explain why a single destructive ALTER shipped in one release with the code would cause an outage.
Use expand/contract: add the new column additively and backfill, deploy code that writes both old and new and reads new-with-fallback, then drop the old column only after every instance runs the new code. Each step stays backward-compatible, so old and new versions share one schema during the rolling deploy that drains and replaces instances. A single destructive ALTER would break the still-running old code and cause an outage.
Common mistakes
- ✗Shipping a destructive ALTER in the same release as the code that reads the column
- ✗Assuming old and new app versions cannot safely share one schema during a rollout
- ✗Dropping the old column before every instance runs the new code
Follow-up questions
- →Why must the contract (drop-old-column) step wait until no instance reads it?
- →How does the load balancer's connection draining avoid dropping in-flight requests?
