Running an A/B test is not just about flipping a switch. It requires method, transparent communication, and collaboration between teams. The goal of a structured process? Reduce "time to insight" while keeping quality high.
In my experience overlooking tests from 30+ product teams, having a clear workflow is the single most impactful thing you can do to scale experimentation. Without it, teams reinvent the wheel every time, make avoidable mistakes, and lose institutional knowledge.
If you are scaling an experimentation program, keep this in mind: every step described below needs to be enforced, not suggested. Playbooks that exist as PDFs nobody reads don't work. These steps should be embedded in your tools and workflows so they cannot be skipped. I advise tracking a process quality score per team and running retrospectives every 3 months to assess whether the process itself needs to evolve.
This is where most experiments are won or lost. Before writing a single line of code, you need:
A well-formed hypothesis. Use this template: "We believe that by changing [specific element], we will increase [goal metric] by [specific percentage] without impacting [invariant metric]."
A hypothesis without a clear metric and a falsifiable prediction is not a hypothesis. It's a wish. If you're scaling, consider using an AI-assisted hypothesis evaluation tool to catch obvious gaps before they reach the review stage (see 🤖 AI tools for experimentation).
Test duration estimation. Use a power calculator to determine how long your test needs to run. Inputs: baseline conversion rate, minimum detectable effect, daily traffic. The industry standard expected lift is 5%, but smaller lifts (1-2%) require significantly larger sample sizes. Tests with higher baseline rates reach significance faster.
A/B Test Sample Size Calculator
Guardrail metrics. Define what you will monitor to ensure the test doesn't cause unintended harm. For e-commerce, booking conversion rate (BCR) drop thresholds are common: 5% for bold tests, 2% for safe, 1% for extra safe. Set these before you start, not after you see results. For a deeper look at setting up guardrails from day one, see the Experimentation governance rollout policy page.
A review meeting with your experimentation expert. This can be the central experimentation team, or in a Center of Excellence model, your local experimentation champion. Align on hypothesis quality, metric definitions, targeting, and traffic allocation. This 30-minute meeting saves weeks of wasted effort. In a champion-based model, this meeting is part of the champion's 20% time allocation.
<aside> 💡
Key takeaway: If you cannot clearly articulate what you will learn from this test regardless of the outcome, you are not ready to run it.
</aside>
Before any test goes live, communicate. No surprises.
This step is not optional. I advise setting up a dedicated communication channel (Slack channel, Teams group, whatever your stack supports) where every test launch is announced with: hypothesis, targeted pages/segments, expected duration, and owner.
Ron Kohavi advocates for automated metric notifications: if a test is designed to impact a specific metric, anyone enrolled to monitor that metric should receive an automatic alert when the test launches. If your experimentation stack supports this, enable it. It prevents the "why did my metric move?" panic that derails teams.
At scale with 50+ concurrent tests, this transparency is not a nice-to-have. It's how you avoid teams stepping on each others toes