Path: Foundation (all roles) | Time: 15 min
<aside> 💡
TL;DR: A results page shows many numbers. Only a few of them matter for your decision. This module teaches you how to read experiment results regardless of your role: what statistical significance actually means, what it does not mean, and how to go from numbers to action.
</aside>
When an experiment completes, the results page shows several pieces of information for each metric.
This is the observed difference between treatment and control. For example: "Conversion rate increased by 1.8 percentage points." The point estimate is your best guess at the true effect. But it is a guess, calculated from a sample.
The confidence interval tells you the range within which the true effect plausibly lies. A 95% CI of [0.5%, 3.1%] means: if we repeated this experiment many times, 95% of the intervals we calculate would contain the true effect.
Three things determine the width: sample size (more users = narrower), metric variance (high variability = wider), and confidence level (99% CI is wider than 95%).
A result is statistically significant when the confidence interval does not include zero (for a two-tailed test) or when the p-value is below the pre-specified alpha level (typically 0.05).
<aside> ⚠️
What significance does NOT mean:
It does not mean the effect is large. A 0.01% increase can be significant with enough users.
It does not mean you should ship. The effect might be real but too small to justify maintenance cost.
It does not mean the opposite of significant is "no effect." A non-significant result means insufficient evidence.
</aside>
The p-value is the probability of observing a result as extreme as yours if the treatment had no effect at all. Smaller p-values = stronger evidence. The threshold (alpha) is set before the experiment. You do not need to memorize the exact definition. What matters: low p-value = effect is probably real.
Before interpreting any metric, check that the experiment ran correctly.
If you assigned 50/50 traffic and observe 48,000 vs 52,000, something might be wrong. An SRM test detects whether the imbalance is larger than chance would explain. If SRM is detected, stop interpreting results. Common culprits: bot filtering, redirect failures, consent flows, caching issues.
Is traffic flowing into all variants as expected? Are any guardrail metrics showing significant deterioration? If a guardrail is breaching, the experiment may need to be stopped regardless of the primary metric result.