Quality Management
The QA-lead toolkit — defect density and other metrics, escaped defects, bug triage, test summary reports, quality gates in CI/CD, MTTD and MTTR, root-cause analysis, risk-based prioritization, and the release go/no-go decision.
12 questions
JuniorTheoryVery commonHow do the incident-response metrics MTTD and MTTR differ, and what does each measure?
How do the incident-response metrics MTTD and MTTR differ, and what does each measure?
MTTD, mean time to detect, is the average from a defect or incident starting until it is noticed. MTTR, mean time to restore, runs from detection until service is back to normal. MTTD grades monitoring and testing; MTTR grades response. Different fixes move them, so track them separately.
Common mistakes
- ✗Merging MTTD and MTTR into one 'incident time' number
- ✗Believing a faster deploy pipeline shortens detection time
- ✗Reading MTTR as time to find the root cause rather than to restore service
Follow-up questions
- →Which of the two would you attack first on a system with weak monitoring?
- →How can MTTR look excellent while users still sit through long outages?
JuniorTheoryCommonWhat is defect density, how do you calculate it, and what does the number tell you?
What is defect density, how do you calculate it, and what does the number tell you?
Defect density is confirmed defects divided by component size — usually per KLOC (thousand lines of code) or per function point. It is comparative: it ranks modules or releases and shows where defects cluster. A high value alone proves nothing — it can mean fragile code or just deeper testing.
Common mistakes
- ✗Reading a high density as proof of bad code rather than of thorough testing
- ✗Comparing density across products that measure size in different units
- ✗Treating density as an absolute quality verdict instead of a comparative signal
Follow-up questions
- →Why can a module with zero reported defects be the riskiest thing you ship?
- →What size unit would you use when lines of code are not comparable?
MiddleTheoryCommonWhat happens in a bug triage meeting, and when is closing a defect as 'won't fix' the right call?
What happens in a bug triage meeting, and when is closing a defect as 'won't fix' the right call?
Triage is the meeting where the lead, product owner and a developer review defects and give each a severity, priority and owner — what ships now, waits, or is rejected. 'Won't fix' is right when the fix's cost or risk outweighs its value: cosmetic, correct behaviour, a duplicate, or a retiring feature. It is a recorded decision, not a way to hide bugs.
Common mistakes
- ✗Rejecting a bug as 'won't fix' just because the sprint is full
- ✗Deciding on fix effort alone, ignoring the defect's value to users
- ✗Treating triage as a solo tester activity rather than a cross-role decision
Follow-up questions
- →Who should have the final say when QA and the product owner disagree in triage?
- →How do you keep a 'won't fix' backlog from quietly accumulating real risk?
MiddleTheoryCommonWhat is defect leakage across phases, and how does defect-detection % differ from pass rate?
What is defect leakage across phases, and how does defect-detection % differ from pass rate?
Defect leakage is a defect slipping from the phase that should have caught it into a later one — worst case, into production. Defect-detection percentage is your net: in-phase ÷ (in-phase + leaked). Pass rate is the share of cases that passed — a different axis. A 99% pass rate with heavy leakage runs green while missing real defects.
Common mistakes
- ✗Treating a high pass rate as proof that few defects escaped
- ✗Counting a developer-caught bug as leakage instead of an early catch
- ✗Confusing leaked-to-production defects with defects merely still open
Follow-up questions
- →Two teams both report a 98% pass rate — how do you tell which tests better?
- →Why does a defect caught in the same phase cost far less than a leaked one?
MiddleDesignCommonYou are chairing the go/no-go meeting for tomorrow's release. Around the table are the product owner, a developer and the release manager, and everyone wants your recommendation. The build has passed most testing but a few defects are still open, and one large module was refactored this cycle. Walk through how you reach a defensible go or no-go verdict. Cover which exit criteria you check, how you assess the regression risk of the refactored area, how you distinguish a known issue you can ship with — a documented workaround — from a blocking defect that must stop the release, and how you would keep a short risk register of the accepted risks with an owner and mitigation for each, so that shipping with open defects is a conscious, recorded decision rather than an accident.
You are chairing the go/no-go meeting for tomorrow's release. Around the table are the product owner, a developer and the release manager, and everyone wants your recommendation. The build has passed most testing but a few defects are still open, and one large module was refactored this cycle. Walk through how you reach a defensible go or no-go verdict. Cover which exit criteria you check, how you assess the regression risk of the refactored area, how you distinguish a known issue you can ship with — a documented workaround — from a blocking defect that must stop the release, and how you would keep a short risk register of the accepted risks with an owner and mitigation for each, so that shipping with open defects is a conscious, recorded decision rather than an accident.
Start from the exit criteria: critical-path coverage, no blockers, defects triaged. Weight regression risk toward the refactored module — it changed, so likeliest to break. A known issue with a workaround and low impact can ship; a blocker — data loss, a broken path, no workaround — is a hard no-go. Log each accepted risk with an owner, so the verdict is defensible.
Common mistakes
- ✗Treating every open defect as a blocker, or every one as shippable
- ✗Not weighting regression risk toward the code that actually changed
- ✗Shipping with open defects but recording no accepted-risk register
Follow-up questions
- →What single open defect would flip your go to a no-go, and why?
- →Who owns an accepted risk once you ship, and what do they commit to?
MiddleDesignCommonA defect that reached production has just been fixed, and you are running the retrospective on it. The team's instinct is to congratulate the developer for the quick patch and move on. You want to prevent the whole class of defect instead. Walk through how you would run a root-cause analysis: how you separate the symptom from the underlying cause, how a technique such as '5 Whys' drills from the visible failure down to the systemic reason, how classifying the defect by type helps you see whether this is a one-off or a pattern across recent escapes, and how the retrospective turns the finding into a concrete preventive action rather than a note nobody reads. Explain what makes the outcome durable instead of a repeat next month.
A defect that reached production has just been fixed, and you are running the retrospective on it. The team's instinct is to congratulate the developer for the quick patch and move on. You want to prevent the whole class of defect instead. Walk through how you would run a root-cause analysis: how you separate the symptom from the underlying cause, how a technique such as '5 Whys' drills from the visible failure down to the systemic reason, how classifying the defect by type helps you see whether this is a one-off or a pattern across recent escapes, and how the retrospective turns the finding into a concrete preventive action rather than a note nobody reads. Explain what makes the outcome durable instead of a repeat next month.
Separate the symptom the user saw from the root cause. Drill with '5 Whys' to a systemic cause — a missing test, a review gap — not the buggy line. Classify the defect and compare recent escapes to tell a one-off from a pattern. The retro's real output is a concrete, owned preventive action that stops the class recurring, not praise for the patch.
Common mistakes
- ✗Stopping at the buggy line instead of the systemic reason it survived
- ✗Turning '5 Whys' into a hunt for a person to blame
- ✗Ending the retro with a note but no owned, concrete preventive action
Follow-up questions
- →How do you avoid '5 Whys' stopping at a convenient but shallow answer?
- →What would tell you an escape is a pattern rather than an isolated miss?
MiddleDesignOccasionalYour team wants to add an automated quality gate to the CI/CD pipeline so that a build cannot be promoted to staging unless it meets an objective quality bar, instead of relying on someone eyeballing the results. Design that gate. Decide which conditions it should enforce — test pass thresholds, code coverage, static-analysis findings, no new high-severity defects — and, crucially, where you draw each line so the gate blocks genuinely risky builds without failing on noise. Explain how you keep it from becoming either a rubber stamp everyone ignores or a flaky blocker that teams learn to bypass, and how metrics such as escaped-defect trends and detection time would tell you whether the gate is actually improving quality over the following months.
Your team wants to add an automated quality gate to the CI/CD pipeline so that a build cannot be promoted to staging unless it meets an objective quality bar, instead of relying on someone eyeballing the results. Design that gate. Decide which conditions it should enforce — test pass thresholds, code coverage, static-analysis findings, no new high-severity defects — and, crucially, where you draw each line so the gate blocks genuinely risky builds without failing on noise. Explain how you keep it from becoming either a rubber stamp everyone ignores or a flaky blocker that teams learn to bypass, and how metrics such as escaped-defect trends and detection time would tell you whether the gate is actually improving quality over the following months.
Define the gate as objective pass/fail checks on every build: a pass threshold, a rising coverage floor, no new high-severity defects, static limits. Draw each line to catch real risk without noise — quarantine flaky tests, gate on new findings not the legacy backlog. Act on failures to stay credible, and watch escaped-defect trends and detection time to confirm quality.
Common mistakes
- ✗Setting absolute thresholds (100% pass, 100% coverage) that fail on noise
- ✗Gating on the whole legacy defect backlog instead of only new findings
- ✗Never measuring whether the gate actually reduces escaped defects over time
Follow-up questions
- →How do you keep flaky tests from making the gate untrustworthy?
- →What trend would prove the gate is helping rather than just slowing merges?
MiddleDesignOccasionalThe regression window for a release has just been cut from three days to one, and the full suite needs three days to run. The date will not move, and the release touches the checkout flow plus a handful of unrelated bug-fixes. You cannot run everything. Walk through how you decide, using a risk-based approach, which tests to run first and which to defer. Explain how you rank areas by risk, what signals — change scope, business impact, past defect history — feed that ranking, how you would use defect metrics from previous releases to find the fragile modules, and how you communicate the residual risk of everything you chose not to run so that the go/no-go decision is made with eyes open rather than blind.
The regression window for a release has just been cut from three days to one, and the full suite needs three days to run. The date will not move, and the release touches the checkout flow plus a handful of unrelated bug-fixes. You cannot run everything. Walk through how you decide, using a risk-based approach, which tests to run first and which to defer. Explain how you rank areas by risk, what signals — change scope, business impact, past defect history — feed that ranking, how you would use defect metrics from previous releases to find the fragile modules, and how you communicate the residual risk of everything you chose not to run so that the go/no-go decision is made with eyes open rather than blind.
Rank each area by risk = likelihood × impact. Impact is highest on business-critical paths like checkout; likelihood is highest where code changed and where defect metrics flag fragile modules. Run that intersection first, defer stable low-impact areas, then state the residual risk you skipped so go/no-go is informed, not blind.
Common mistakes
- ✗Prioritizing by test speed or count instead of by risk to the business
- ✗Ignoring past defect history when picking the fragile areas to cover
- ✗Silently dropping deferred areas instead of surfacing the residual risk
Follow-up questions
- →How would you factor in an area that is stable but business-critical, like login?
- →What do you do when two high-risk areas both need more time than remains?
MiddleTheoryOccasionalWhat is a test summary report, what does it contain, and who reads it?
What is a test summary report, what does it contain, and who reads it?
A test summary report closes a test cycle. It states what was tested and what was left out, how many cases passed, failed and were blocked, open defects by severity, and metrics like defect density and pass rate — ending with an explicit release recommendation. It leads with the verdict for the lead and stakeholders, not raw logs.
Common mistakes
- ✗Dumping raw logs instead of leading with a clear release recommendation
- ✗Omitting what was NOT tested, hiding the residual risk from readers
- ✗Writing it for testers rather than for the lead and stakeholders
Follow-up questions
- →Why is the 'what we did not test' section as important as the pass rate?
- →How would the report change for a hotfix versus a major release?
SeniorDesignOccasionalIt is the afternoon before a release that has been announced to customers, and you have just found a critical bug — a genuine data-corruption path on a common user action. The developers say a proper fix needs two days; a rushed patch could be attempted tonight but carries its own risk. Marketing has already promised the date. Walk through how you handle this, not just technically but as the person the go/no-go now hinges on. Cover how you confirm and quantify the bug's real impact and how often it triggers, how you frame the options — slip the date, ship with the feature disabled, attempt the risky hotfix — for the decision-makers, how you resist pressure to simply pass it, and what you insist on before any 'ship anyway' path is taken so the business decides with full knowledge.
It is the afternoon before a release that has been announced to customers, and you have just found a critical bug — a genuine data-corruption path on a common user action. The developers say a proper fix needs two days; a rushed patch could be attempted tonight but carries its own risk. Marketing has already promised the date. Walk through how you handle this, not just technically but as the person the go/no-go now hinges on. Cover how you confirm and quantify the bug's real impact and how often it triggers, how you frame the options — slip the date, ship with the feature disabled, attempt the risky hotfix — for the decision-makers, how you resist pressure to simply pass it, and what you insist on before any 'ship anyway' path is taken so the business decides with full knowledge.
First confirm it: reproduce it and quantify impact and trigger rate — data corruption on a common action is severe. Do not silently pass it under pressure; that is the one thing you cannot do. Frame options, don't decide alone: slip, disable the feature, or risk the hotfix. Any 'ship anyway' path is a documented, owned decision made with full knowledge of the risk.
Common mistakes
- ✗Passing a data-corruption bug as a known issue under schedule pressure
- ✗Applying a risky hotfix quietly without escalating or quantifying impact
- ✗Making the go/no-go call alone instead of framing options for the business
Follow-up questions
- →How do you push back when a manager pressures you to just pass it?
- →What makes 'ship with the feature disabled' safer than the rushed hotfix here?
SeniorDesignOccasionalThe day before a release you have to send a short written status update to non-technical stakeholders — a product director and two business owners who will not read a bug tracker and do not know what 'p2 regression' or 'flaky suite' means. They need to make a business decision from your message. Describe how you would write that update. Cover how you lead with the bottom line — ship, hold, or ship-with-caveats — rather than burying it under detail, how you translate defect counts and severities into business impact and user-facing risk they can weigh, how you make the residual risk and any assumptions explicit and honest without technical jargon, and how you keep it short enough that a busy executive actually reads it and can act on it.
The day before a release you have to send a short written status update to non-technical stakeholders — a product director and two business owners who will not read a bug tracker and do not know what 'p2 regression' or 'flaky suite' means. They need to make a business decision from your message. Describe how you would write that update. Cover how you lead with the bottom line — ship, hold, or ship-with-caveats — rather than burying it under detail, how you translate defect counts and severities into business impact and user-facing risk they can weigh, how you make the residual risk and any assumptions explicit and honest without technical jargon, and how you keep it short enough that a busy executive actually reads it and can act on it.
Lead with the bottom line — ship, hold, or ship-with-caveats — so the reader gets the decision first, before any detail. Translate defects into business terms: not 'three p2 bugs' but 'checkout works, but discount codes fail for ~5% of users, with a workaround'. Drop the jargon, state the residual risk and assumptions honestly, and end with the one decision you ask for.
Common mistakes
- ✗Burying the ship/hold decision at the end instead of leading with it
- ✗Leaving defect counts and severities untranslated into business impact
- ✗Hiding the residual risk to look reassuring, or omitting the ask
Follow-up questions
- →How do you convey bad news honestly without triggering panic or blame?
- →What would you cut first if the update had to fit in three sentences?
SeniorDesignOccasionalLeadership asks you to prove that the QA team is actually worth its cost, and warns you off vanity numbers like 'test cases written' or 'tests executed', which measure activity rather than value. Design a way to measure how effective your testing really is. Explain which signals genuinely reflect effectiveness — how many defects your testing catches before release versus how many escape to users, how thoroughly it exercises the real risk rather than just lines of code, and what those numbers cost in time. Cover why you must read trends over several releases rather than a single absolute figure, how you would guard against a metric that can be gamed, and how you would present the result so it drives improvement instead of blame.
Leadership asks you to prove that the QA team is actually worth its cost, and warns you off vanity numbers like 'test cases written' or 'tests executed', which measure activity rather than value. Design a way to measure how effective your testing really is. Explain which signals genuinely reflect effectiveness — how many defects your testing catches before release versus how many escape to users, how thoroughly it exercises the real risk rather than just lines of code, and what those numbers cost in time. Cover why you must read trends over several releases rather than a single absolute figure, how you would guard against a metric that can be gamed, and how you would present the result so it drives improvement instead of blame.
There is no single number. Read effectiveness as defects caught before release versus those escaping to users — defect detection percentage — plus how well testing hits the real risk, not just coverage. Track trends across releases, guard against gaming: activity counts inflate without catching more. Present it to drive prevention, not blame.
Common mistakes
- ✗Equating activity counts (cases written, tests run) with effectiveness
- ✗Reading a single absolute figure instead of a trend across releases
- ✗Reporting effectiveness metrics in a way that ranks or blames people
Follow-up questions
- →How would you detect that a team is gaming the metric you chose?
- →Why is escaped-defect rate more honest than any in-house pass rate?
