Path: Product Manager | Time: 12 min
<aside> 💡
TL;DR: One primary success metric. As many guardrails as you need. Zero metrics chosen after seeing results. This module teaches the discipline that separates reliable experimentation from data-driven storytelling.
</aside>
Every experiment needs exactly one primary success metric. Not two. Not five. One.
Every additional primary metric increases your false positive risk. If you test five metrics at alpha = 0.05, the probability that at least one shows a false positive is roughly 23%. You can correct for this, but corrections reduce power. The solution: choose the one metric that most directly answers your hypothesis.
A good success metric has four properties:
Sensitivity. It responds to the type of change you are making. A metric measured close to the point of treatment is more sensitive than one measured far downstream.
Directionality. You know which direction is good. If "time on page" could mean engagement or confusion, it is a poor primary metric.
Robustness. Not dominated by outliers. Revenue per user is sensitive to a few high-value transactions. Consider capped metrics, medians, or conversion rates.
Coverage. It captures the effect for the population you are testing. Match the metric scope to the treatment scope.
| Product area | Common primary metrics | Watch out for |
|---|---|---|
| Booking/checkout | Completion rate, funnel step conversion | Metrics too far downstream from the change |
| Search/discovery | Click-through rate, selection rate | CTR can be gamed by clickbait; pair with conversion |
| Pricing/ancillaries | Attach rate, revenue per visitor, take rate | Revenue is high variance; consider capping |
| Engagement | Sessions per user, actions per session, return rate | "Time on site" is ambiguous |
| Retention | 7-day/30-day retention, churn rate | Needs long experiments; may not be feasible as primary |
Guardrails protect what you cannot afford to degrade. They use a non-inferiority test: you are not asking "did load time improve?" but "did load time stay within an acceptable range?"
The non-inferiority margin (NIM) is the maximum degradation you accept. Example: "Page load time must not increase by more than 200ms." Too tight = alert fatigue. Too loose = real harm. The right NIM is the largest degradation that would not materially affect the user experience.
Secondary metrics are exploratory. They help you understand the mechanism behind your result. If the primary improves and a secondary shows an interesting pattern, that is a signal for a future experiment. But you do not make the ship/no-ship decision on secondaries.
HARKing (Hypothesizing After Results are Known) happens when you retroactively promote a secondary metric to primary because your original primary did not move. With enough metrics, one will be significant by chance. Retroactively promoting it is dressing up a false positive as a validated result.