PM1. Designing your experiment

Path: Product Manager | Time: 15 min

<aside> 💡

TL;DR: Designing an experiment is not "set it up and let it run." You need to answer three questions before anything goes live: What am I testing? Can I actually test it? And which test design fits this situation? Get these wrong and you waste weeks.

</aside>

From business question to test design

You have a hypothesis from F3. Now you need to turn it into an experiment that will actually produce a reliable answer.

Question 1: Is this testable?

Not everything can be A/B tested. Before designing the experiment, check:

Can you randomize? Standard A/B tests require random assignment of individual users to variants. If your change applies to everyone on a route, a region, or a market simultaneously, you need a different design (switchback, quasi-experimental).
Is the change isolatable? Can you show the treatment to one group without it affecting the control group?
Do you have enough traffic? The smaller the effect you want to detect, the more traffic you need. A test that needs 500,000 users per variant on a page that gets 10,000 visitors per week will run for almost a year.

Question 2: Which test design fits?

🖼️ IMAGE PLACEHOLDER

The Experimentation Compass: decision tree starting with "Can you randomize at the individual user level?" branching into A/B test, switchback, quasi-experimental, and pre/post designs. Adapt from Experimentation Compass JPG.

Design	When to use	Traffic needed	Duration
A/B test	User-level randomization possible, enough traffic	High (depends on MDE)	2-4 weeks
A/B/n test	Compare 3+ variants simultaneously	Higher (split across variants)	2-4 weeks
Switchback	Cannot randomize at user level (pricing, marketplace)	Medium	4-8 weeks
Quasi-experimental	Cannot randomize at all (regulatory, infrastructure)	Varies	Varies

As a PM, you do not need to design switchback or quasi-experimental methods yourself. But you need to know they exist, so you do not abandon testing when standard A/B is not possible.

Question 3: How long will it run?

Test duration is a function of traffic volume, MDE, baseline metric value, and number of variants. Your data scientist runs the calculation. Your job is to set the MDE: what is the smallest effect worth shipping?

A good rule of thumb: if the calculated duration is longer than 4 weeks, reconsider. Either increase the MDE, increase traffic allocation, or consider whether this is the right test to run right now.

Setting up the experiment

Audience. Who sees this experiment? Define inclusion and exclusion criteria.

Traffic allocation. 50/50 gives maximum statistical power. Smaller treatment allocations (e.g., 10/90) limit risk but increase duration.