Advanced Test Design
Test-design techniques beyond partitions and boundaries — state transition and use-case testing, pairwise combinatorics, error guessing, the test oracle, code coverage versus test coverage, and designing a system to be testable.
9 questions
JuniorTheoryVery commonWhat is a test oracle, and what can serve as one in practice?
What is a test oracle, and what can serve as one in practice?
A test oracle is whatever tells you the observed result is correct. It can be the specification, a reference implementation, a previous version, an invariant the result must satisfy, or a human judgement. Without an oracle a test can only show the system did not crash, not that it did the right thing.
Common mistakes
- ✗Confusing the oracle with the tool that runs the test
- ✗Assuming only a written specification can be an oracle
- ✗Calling a crash-free run a pass with no oracle at all
Follow-up questions
- →What is the oracle problem, and when does it bite hardest?
- →How can an invariant serve as an oracle without an exact expected value?
MiddleTheoryVery commonWhat is state transition testing, and what does it cover that partitions miss?
What is state transition testing, and what does it cover that partitions miss?
State transition testing models the system as states, events and transitions, then derives cases from the diagram. You test valid transitions, the expected resulting state, and — crucially — invalid events fired in a state that should reject them. It catches order-dependent and history-dependent defects that equivalence partitioning, treating each input alone, never sees.
Common mistakes
- ✗Testing only valid transitions, never invalid events in a state
- ✗Ignoring the resulting state and checking only that no error appeared
- ✗Applying it to independent inputs where order carries no meaning
Follow-up questions
- →How do you decide when a state model is worth building for a feature?
- →What extra cases does an invalid-transition sweep add over the happy path?
JuniorTheoryCommonWhat is error guessing, and what does a tester base the guesses on?
What is error guessing, and what does a tester base the guesses on?
Error guessing is an experience-based technique: the tester predicts where defects are likely and designs cases straight at them. The guesses come from past defects, known weak spots, developer habits and typical traps — empty input, zero, nulls, duplicates. It complements the formal techniques and never replaces them.
Common mistakes
- ✗Treating error guessing as random input with no basis in experience
- ✗Using it as a substitute for equivalence classes and boundary analysis
- ✗Never feeding real past defects back into the guesses
Follow-up questions
- →Which past artefacts would you mine to sharpen your guesses?
- →Why can error guessing not be the only technique on a release?
MiddleTheoryCommonWhat is pairwise testing, and why does it cut the number of combinations?
What is pairwise testing, and why does it cut the number of combinations?
Pairwise (2-way combinatorial) testing covers every pair of parameter values in at least one case, instead of every full combination. It rests on the empirical finding that most defects are triggered by one or two interacting factors, so testing all pairs finds them with a tiny fraction of the exhaustive set. It misses genuine three-way interactions.
Common mistakes
- ✗Expecting pairwise to catch three-way and higher interactions
- ✗Confusing it with random sampling of two values per field
- ✗Thinking it reproduces exhaustive coverage at lower cost
Follow-up questions
- →When would you step up from pairwise to 3-way coverage?
- →How do constraints between parameters change the generated set?
MiddleTheoryCommonWhat is use-case-based testing, and how does it derive test cases?
What is use-case-based testing, and how does it derive test cases?
Use-case-based testing derives cases from actor–system interactions written as use cases. You test the main success scenario end to end, then every alternate and exception flow the use case lists. Because it follows user goals across components, it finds integration and workflow defects that isolated component tests miss.
Common mistakes
- ✗Testing only the main success scenario, skipping alternate and exception flows
- ✗Confusing a use case with a single UI screen or component
- ✗Losing the actor's goal and testing steps in isolation
Follow-up questions
- →How do you turn an alternate flow into an independent test case?
- →Why does this technique surface integration defects unit tests miss?
JuniorTheoryOccasionalHow does code coverage differ from test coverage, and what do its levels mean?
How does code coverage differ from test coverage, and what do its levels mean?
Test coverage asks how much of the requirements or risk your cases exercise; code coverage asks how much of the source they run. Code coverage has levels: statement (each line hit), branch (each true/false edge taken), and path (each route through the code). High code coverage does not imply the behaviour was actually checked.
Common mistakes
- ✗Conflating code coverage with requirements/test coverage
- ✗Reading 100% statement coverage as proof the behaviour is correct
- ✗Confusing branch coverage with statement coverage
Follow-up questions
- →Why can a suite hit 100% statement coverage yet miss a branch?
- →Why does path coverage explode combinatorially on real code?
MiddleDesignOccasionalYour team is starting a new order-management service and brings you in at design time, before any code exists. The lead asks how QA can make the system testable from the start rather than bolting testing on later. Working with the developers, what design-for-testability properties would you push for — around observability, control, isolation, and determinism — and how would each one make defects cheaper to find? Explain how you would negotiate these with developers who are focused on shipping features, and how you would tell whether the resulting system actually is more testable.
Your team is starting a new order-management service and brings you in at design time, before any code exists. The lead asks how QA can make the system testable from the start rather than bolting testing on later. Working with the developers, what design-for-testability properties would you push for — around observability, control, isolation, and determinism — and how would each one make defects cheaper to find? Explain how you would negotiate these with developers who are focused on shipping features, and how you would tell whether the resulting system actually is more testable.
Push for observability (logs, metrics, health, state), controllability (hooks, feature flags, seedable clock), isolation (stub dependencies at a seam), and determinism (no hidden time or randomness). Each shrinks the gap between a defect and an observable symptom, so failures are reproducible. Sell each hook as faster feedback, and judge success by falling defect-detection time.
Common mistakes
- ✗Treating testability as QA-only, with no design-time input
- ✗Equating testability with a code-coverage number alone
- ✗Adding test back doors that skip real validation or invariants
Follow-up questions
- →Which single testability hook gives the best return on the earliest feature?
- →How would you stop a control hook from weakening production security?
SeniorDesignOccasionalYour product adds an ML-powered support-reply suggester: given a customer message it returns a suggested answer plus a confidence score. The same input can yield different wording between model versions, and even identical inputs may vary slightly at runtime, so there is no single expected string to assert. The team still needs a regression signal that catches quality drops before release. How would you design the testing approach when a classic exact-match oracle does not exist — what kinds of oracle you would substitute, how you would keep the suite stable against acceptable variation, how you would sample the huge input space, and what you would gate a release on?
Your product adds an ML-powered support-reply suggester: given a customer message it returns a suggested answer plus a confidence score. The same input can yield different wording between model versions, and even identical inputs may vary slightly at runtime, so there is no single expected string to assert. The team still needs a regression signal that catches quality drops before release. How would you design the testing approach when a classic exact-match oracle does not exist — what kinds of oracle you would substitute, how you would keep the suite stable against acceptable variation, how you would sample the huge input space, and what you would gate a release on?
Replace the exact-match oracle with weaker ones: invariants (no PII leak, confidence in range), metamorphic relations (paraphrased input keeps intent), and a golden set scored by similarity, not string equality. Stabilize by asserting on properties and thresholds, not wording. Sample the space by risk-weighted pairwise combinations. Gate on aggregate metrics, not a single reply.
Common mistakes
- ✗Forcing an exact-match oracle onto non-deterministic output
- ✗Concluding the feature is untestable because output varies
- ✗Gating on a single reply instead of aggregate quality metrics
Follow-up questions
- →Give a metamorphic relation you would assert on this suggester.
- →How would you set the release threshold without overfitting to the golden set?
MiddleDesignRareAn e-commerce checkout runs as four steps — cart, address, payment, confirmation — and the server expires an idle session after 15 minutes. A shopper can move forward, jump back to edit an earlier step, or reload, and the timeout can fire at any point, including mid-payment. Reserved stock is released when the session dies. Design the testing approach for how the flow behaves as state and time interact: which states and transitions you would model, what must happen when the timeout fires at each step, and how you would verify no order is charged or confirmed against an expired session and no stock is double-reserved or lost.
An e-commerce checkout runs as four steps — cart, address, payment, confirmation — and the server expires an idle session after 15 minutes. A shopper can move forward, jump back to edit an earlier step, or reload, and the timeout can fire at any point, including mid-payment. Reserved stock is released when the session dies. Design the testing approach for how the flow behaves as state and time interact: which states and transitions you would model, what must happen when the timeout fires at each step, and how you would verify no order is charged or confirmed against an expired session and no stock is double-reserved or lost.
Model each step as a state with events for next, back, reload and timeout, and derive cases from that diagram. For each step, test that a fired timeout lands the user in a safe expired state, releases the reservation exactly once, and blocks any later charge. Read correctness from server-side session and order state, not the UI, with stock neither double-reserved nor lost.
Common mistakes
- ✗Testing only the forward happy path, never a timeout mid-flow
- ✗Reading pass/fail from the UI instead of server session and order state
- ✗Ignoring stock reservation — double-reserve or lost stock on expiry
Follow-up questions
- →How would you test a timeout that fires exactly during the payment call?
- →What server-side evidence proves an expired session cannot be charged?
