Distributed Systems Testing

Assert on the eventual state, not on timing: poll the callback endpoint or a status field with a bounded timeout and back-off instead of a fixed sleep. Drive each branch deliberately — a clean file, an infected file the scan must reject, and a forced step failure so the cron retry runs — and verify the webhook payload, the email, and idempotency when a retry re-delivers. Capture the callback with a stub receiver so the outcome is observable.

Verification shifts from a one-time gate to a live comparison: watch the new version's error rate, latency, and key business metric against the old baseline serving real traffic, not a fixed threshold alone. Define the rollback trigger up front — a bounded regression on those signals auto-reverts or drains the canary. Keep the comparison honest by splitting cohorts randomly and comparing like-for-like traffic, so a metric shift reflects the code, not a skewed audience or time of day.

Test that processing is idempotent: re-deliver the same bid and assert the outcome is unchanged — no double bid, no double charge. Pin down the real ordering guarantee: a queue orders only within a partition or key, not globally, so route bids for one item to one key and assert per-item order while accepting that unrelated bids interleave. Sign off only when duplicates are provably absorbed and the ordering the product needs matches what the partitioning enforces.

Neither extreme. Push the bulk down: unit tests per service for logic, contract tests at every seam so each pair stays compatible without deploying everything, and a thin e2e tier over only the critical journeys — a real checkout, a declined payment, an out-of-stock item. Each level owns a distinct failure class and the pyramid keeps feedback fast. Reserve e2e for the few paths where a cross-service interaction, not one service's logic, is the actual risk.

Inject the failure modes deliberately — latency, errors, and full outage on each dependency, plus killing one inventory node mid-request. Assert the page degrades rather than breaks: with recommendations down it drops that block and still renders the product; when the service is merely slow, a timeout and fallback keep the page inside its budget instead of hanging. For the node kill, assert a committed write survives and the request either retries or fails cleanly — no partial or lost data.

A Pact contract test pins one consumer–producer pair: the consumer records the requests and responses it expects, the producer replays them, and each side is verified alone without both deployed. End-to-end runs the whole chain live — it catches wiring and data-flow bugs but is slow and flaky. Contract tests pinpoint which side broke a contract; e2e proves the assembled system works.

First confirm the write is durable, not lost: read from the primary to check it persisted, which rules out data loss. Then measure the convergence window — how long until every replica and the cache serve the new value — and compare it to an agreed staleness bound. It is expected eventual consistency if reads converge inside that bound; it is a real bug if the window exceeds the bound, a stale value outlives its cache TTL, or the value never converges.

Reproduce with several concurrent clients, not one: script two or more sessions, inject reconnects and network drops, and interleave sends so the races surface. Assert per-sender ordering, exactly-once display after a reconnect, and that every client converges to the same final list within a bounded window. Separate a real bug — permanent divergence, or messages lost or reordered after convergence — from expected transient lag while the state is still settling.

Test against the vendor's sandbox and a contract-verified stub, not production: the sandbox runs the real protocol with test cards for each outcome, while the stub deterministically drives declines, timeouts, and duplicate or out-of-order webhooks. Keep the double faithful with a contract test against the vendor so it cannot silently drift. Verify settlement by asserting the eventual state your webhook handler reaches — idempotent and order-independent — never a fixed wait.

The seam carries the contract — field names and types, serialisation, status codes, timeouts, auth, and version drift. A unit test stubs the other side, so it only ever confirms your own assumption; an integration test makes the real call and catches that assumption being wrong.