Retention jumped the week you redefined an 'active user' — real gain or definitional artifact?
Retention (and DAU) rose sharply the same week the team changed the definition of an "active user" in the metric query. Nothing else shipped that week. The old and new definitions:
-- OLD: active = performed a core action
WHERE event_type = 'core_action'
-- NEW: active = opened the app (any event)
WHERE event_type IS NOT NULLDetermine whether the retention gain is a real improvement or an artifact of the definition change, and what you would check to tell the two apart. Diagnose the cause.
Almost certainly definitional: widening 'active' from a core action to any app-open inflates every retention cut, so it steps up with no behaviour change. Recompute both weeks on one fixed definition — an artifact makes the jump vanish, while a real gain shows under the old definition.
- ✗Crediting a metric jump that lands exactly on a definition change
- ✗Comparing to an external benchmark instead of recomputing on one definition
- ✗Assuming the broader definition is automatically the more correct one
- →How would versioning metric definitions prevent this confusion?
- →What backfill would let you compare both weeks fairly?
The definition change widens the numerator, so the key check is recomputing both weeks on one definition:
-- Recompute BOTH periods on ONE fixed definition, then compare
SELECT week,
COUNT(DISTINCT user_id) FILTER (WHERE event_type = 'core_action') AS active_old_def,
COUNT(DISTINCT user_id) FILTER (WHERE event_type IS NOT NULL) AS active_new_def
FROM events
GROUP BY week
ORDER BY week;If the retention jump appears in the active_new_def column but not in active_old_def, the gain is a definitional artifact: widening 'active' from a core action to any app-open simply admitted more users into both the numerator and denominator. A real improvement would also show up under active_old_def and would build gradually, not step exactly on the week the query changed.
