Performance Testing
Performance testing as a practice — throughput, latency and response time, establishing a baseline, modelling realistic load in JMeter or k6, soak and endurance runs, vertical versus horizontal scalability, and locating the actual bottleneck.
7 questions
JuniorTheoryVery commonWhat is a performance baseline, and why take one before optimizing anything?
What is a performance baseline, and why take one before optimizing anything?
A baseline is a set of performance numbers — response-time percentiles, throughput, error rate — recorded on a known build, in a known environment, under a known load profile. Without one you cannot tell whether a change helped or hurt: every later run is compared against it, which is what turns a suspected regression into a measurable fact.
Common mistakes
- ✗Optimizing before measuring, with nothing to compare the result against
- ✗Recording a baseline without pinning the build, environment and load profile
- ✗Using the fastest observed response instead of percentiles as the baseline
Follow-up questions
- →What invalidates a baseline and forces you to record a new one?
- →Why must the environment be pinned for a baseline to mean anything?
JuniorTheoryVery commonHow do throughput, latency and response time differ, and why report percentiles?
How do throughput, latency and response time differ, and why report percentiles?
Throughput is work per unit of time — requests per second. Response time is how long one request takes, and latency is the delay before work begins. They are independent axes: throughput can rise while response time collapses. Report percentiles — p95, p99 — never an average, which hides the slow tail that users actually feel.
Common mistakes
- ✗Reporting an average response time instead of the p95 and p99 percentiles
- ✗Treating throughput and response time as the same metric
- ✗Assuming high throughput automatically means fast individual responses
Follow-up questions
- →What does a p99 that is 40 times the median tell you about the system?
- →Why can throughput stay flat while response time keeps climbing?
MiddleTheoryCommonUnder load a system is too slow — how do you find the actual bottleneck rather than guess?
Under load a system is too slow — how do you find the actual bottleneck rather than guess?
The bottleneck is the one resource that saturates first and caps throughput. Drive load up in steps against the baseline while watching every resource — CPU, memory, I/O, the database, thread pool, locks — and where requests queue. It is whatever hits its limit while the others still have headroom; fix it and the bottleneck moves, so re-measure.
Common mistakes
- ✗Guessing from the code instead of watching resources under real load
- ✗Chasing the single slowest request rather than the saturated resource
- ✗Assuming the bottleneck is always the database and skipping the search
Follow-up questions
- →Why does fixing one bottleneck often just move it to the next resource?
- →How do you distinguish a saturated resource from one that is merely busy?
MiddleTheoryCommonHow does testing vertical scalability differ from testing horizontal scalability?
How does testing vertical scalability differ from testing horizontal scalability?
Vertical means a bigger node — more CPU and RAM; horizontal means more nodes behind a balancer. Test both by adding resource in steps and plotting throughput against it, watching where the curve stops being near-linear. Vertical hits a hardware ceiling; horizontal exposes a shared bottleneck — the database, a lock, sticky sessions — that caps the gain.
Common mistakes
- ✗Swapping the definitions — calling scale-out vertical and scale-up horizontal
- ✗Measuring one request's latency instead of throughput versus resource added
- ✗Assuming scaling is always linear and ignoring the shared bottleneck
Follow-up questions
- →What shared resource most often caps horizontal scaling in practice?
- →How would you spot the point where added nodes stop helping throughput?
MiddleDesignOccasionalYour team is about to load-test the checkout API of an online store. You have three months of production access logs and an APM dashboard, so the real traffic mix is known: roughly 70% catalogue reads, 20% search, 8% cart updates, 2% orders, with peaks near 1200 requests per second on weekday evenings and a much sharper spike during the monthly sale. The load environment is a scaled-down copy of production — one third of the application nodes, but the same database. A baseline has already been recorded, and you may use JMeter or k6. Describe how you would model the load so the run says something true about production: what you would take from the logs, how you would shape the arrival of virtual users over time, and what you would compare the result against.
Your team is about to load-test the checkout API of an online store. You have three months of production access logs and an APM dashboard, so the real traffic mix is known: roughly 70% catalogue reads, 20% search, 8% cart updates, 2% orders, with peaks near 1200 requests per second on weekday evenings and a much sharper spike during the monthly sale. The load environment is a scaled-down copy of production — one third of the application nodes, but the same database. A baseline has already been recorded, and you may use JMeter or k6. Describe how you would model the load so the run says something true about production: what you would take from the logs, how you would shape the arrival of virtual users over time, and what you would compare the result against.
Build the profile from the logs, not from a guess: reproduce the endpoint mix and its weights, keep think time and session state so a virtual user behaves like a person, and ramp arrivals in steps to the target rate instead of starting at full load. Scale the target to the smaller environment, then judge the run against the recorded baseline on the same percentiles.
Common mistakes
- ✗Guessing the endpoint mix instead of deriving it from production logs
- ✗Starting at full load, with no ramp and no think time between requests
- ✗Reading raw numbers off a scaled-down environment as if they were production's
Follow-up questions
- →How would you model the monthly sale spike differently from the evening peak?
- →Which numbers from a one-third environment can still be trusted, and which cannot?
MiddleTheoryOccasionalWhat is a soak (endurance) test, and which failures does it catch that a short run misses?
What is a soak (endurance) test, and which failures does it catch that a short run misses?
A soak test holds a moderate, production-like load steady for hours or days. It targets the slow failure a short run never reaches: memory leaks, a pool that never returns handles, disks filling with logs, response time drifting upward. Judge it on resource trends over time, not on one peak number.
Common mistakes
- ✗Confusing a soak test with a stress test — high intensity instead of long duration
- ✗Running it too briefly to let a leak or slow drift accumulate
- ✗Judging it on a peak number instead of resource trends over time
Follow-up questions
- →Which resource graphs would you watch across an overnight soak, and why?
- →How would you tell a genuine memory leak from normal cache warm-up in the trend?
SeniorDesignRareYour checkout flow calls an external payment provider on every order. It is a black box: you cannot instrument it, you have no test tenant, and its contract forbids driving synthetic load at production — hitting it hard risks rate-limit bans, real transaction fees, and degrading service for the provider's other customers. Yet performance sign-off for the release depends on knowing how checkout behaves under peak load, and the payment call sits on the critical path. You have three months of your own logs showing the provider's real latency and error-rate distribution, and its published SLA. Design an approach that lets you load-test and sign off on checkout performance without load-testing the third party itself — say what you would drive load against, how you would represent the provider, and what you would actually measure and report.
Your checkout flow calls an external payment provider on every order. It is a black box: you cannot instrument it, you have no test tenant, and its contract forbids driving synthetic load at production — hitting it hard risks rate-limit bans, real transaction fees, and degrading service for the provider's other customers. Yet performance sign-off for the release depends on knowing how checkout behaves under peak load, and the payment call sits on the critical path. You have three months of your own logs showing the provider's real latency and error-rate distribution, and its published SLA. Design an approach that lets you load-test and sign off on checkout performance without load-testing the third party itself — say what you would drive load against, how you would represent the provider, and what you would actually measure and report.
Don't load-test the provider — replace it with a stub replaying the latency and error-rate distribution your logs and its SLA show, including the slow tail. Drive load only at your own checkout, checking its behaviour around it: a timeout bounds the call, retries back off, and a circuit breaker plus fallback keep it responsive when the provider is slow. Report end-to-end percentiles and degradation; take a real number only within the agreed rate limit or a sandbox.
Common mistakes
- ✗Driving synthetic peak load at the provider's real production endpoint
- ✗Dropping the dependency from the test and assuming it is infinitely fast
- ✗Stubbing only the happy path, never the slow tail or the error responses
Follow-up questions
- →How would you derive the stub's latency and error distribution from your logs and the SLA?
- →What would make you insist on a small, real measurement despite the constraints?
