F3. Hypotheses and metrics

Path: Foundation (all roles) | Time: 15 min

<aside> 💡

TL;DR: A bad hypothesis makes an experiment useless regardless of how well you run it. A bad metric makes a winning result meaningless. This module teaches you how to write hypotheses that are genuinely testable and choose metrics that actually measure what you care about.

</aside>

What makes a hypothesis testable

A hypothesis is not a feature description. "We should add a progress bar to the checkout" is not a hypothesis. It is a solution. A hypothesis connects that solution to a measurable outcome through a causal logic.

The structure is:

If we [specific change] for [defined audience], then [measurable outcome] will [direction of change] by [expected magnitude], because [rationale based on evidence or reasoning].

Every part of this template matters:

Specific change: What exactly are you changing? Not "improve the checkout" but "add a 3-step progress bar to the checkout flow showing current step, next step, and total steps."
Defined audience: Who sees this change? All users? Mobile only? New users in their first 30 days?
Measurable outcome: What metric will you look at? It must be something you can actually measure with your instrumentation.
Direction and magnitude: Are you expecting an increase or decrease? By how much? This sets the Minimum Detectable Effect for your sample size calculation.
Rationale: Why do you believe this change will have this effect? A hypothesis without a rationale is just a guess.

The hypothesis coach

I use a quality scoring framework to evaluate hypotheses before they go into experiment design. Every hypothesis is scored on five dimensions:

Dimension	Question	Score 1 (weak)	Score 5 (strong)
Specificity	Is the change precisely defined?	Vague feature description	Exact UI/logic change specified
Measurability	Can we measure the expected outcome?	No clear metric or metric not instrumented	Primary metric exists, is tracked, and has baseline data
Rationale strength	Is the causal mechanism plausible?	"We think users will like it"	Based on prior experiments, user research, or behavioral theory
Falsifiability	Can the hypothesis be proven wrong?	Vague enough to be "confirmed" by any result	Clear criteria for validation and rejection
Scope	Is the change isolated enough to attribute cause?	Bundles multiple changes that confound each other	Single change, single causal chain

A hypothesis that scores below 3 on any dimension should be rewritten before the experiment is designed. This is not bureaucracy. It is quality control.

Composite hypotheses

Many experiments have multiple metrics. For these, write a composite hypothesis:

Primary hypothesis: "If we add a progress bar to checkout, completion rate will increase by at least 2 percentage points."
Guardrail statement: "Page load time will not increase by more than 200ms (non-inferiority margin)."