Most organizations begin their experimentation journey with genuine enthusiasm. Leadership champions the data-driven approach, teams eagerly embrace A/B testing, and early wins create momentum. Then the plateau hits. Teams run duplicate experiments across departments. Insights remain trapped in individual projects. The connection between test results and business decisions weakens. Only about 12% of organizations rate their experimentation strategy and culture as truly transformative.
The gap between initial excitement and sustained capability is not a motivation problem. It is an infrastructure problem. You cannot train your way to scale. You need systems.
I structure experimentation training in three tiers, each building on the previous one.
Every product team member who touches experimentation needs to understand the basics: what a hypothesis is, what statistical significance means (and does not mean), why you cannot peek at results early, and how to read a test conclusion.
This is not a statistics course. It is a 2-hour session focused on practical literacy. The goal is that anyone in the organization can look at an experiment result and understand whether it is trustworthy. I run this as an onboarding cohort for new teams joining the experimentation program, typically 4-6 teams at a time.
Key topics: what makes a good hypothesis, the difference between statistical and business significance, the peeking problem, how to read confidence intervals, what guardrail metrics are and why they matter.
Champions and data analysts need deeper knowledge. This tier covers hypothesis design using the standard template, power calculations and sample size estimation, test setup and QA, guardrail monitoring, proper documentation, and the full experimentation process (see 🔄 The experimentation process).
This is a hands-on workshop, not a lecture. Teams bring real experiment ideas and work through them end to end during the session. By the end, each team has a fully designed experiment ready to launch, not just theoretical knowledge.
I allocate a full day for this, split into morning (methodology) and afternoon (hands-on design with their actual product).
The central team and experienced champions need advanced topics: one-tailed vs. two-tailed testing (see 🎯 Hypothesis types and when to use one-tailed tests), quasi-experimental methods (see 🧪 When randomization is not possible), multi-armed bandits, Bayesian approaches, interaction effects between concurrent experiments, and how to design decision protocols (see 📋 Decision protocols).
This is ongoing education, not a one-time event. I run monthly deep-dives on specific topics, rotating through the advanced curriculum over the course of a year.
<aside> 💡
Training alone does not change behavior. I have seen organizations run excellent training programs where nothing changed afterward because there was no system to enforce what people learned. Training must be paired with governance (see 🔒 From hope-based to enforced governance). The champion role exists precisely to bridge this gap (see 🏦 The Center of Excellence model).
</aside>
The single most effective enablement activity I have run is the experimentation hackathon. Not a traditional hackathon where people build prototypes. An experimentation hackathon where teams design complete experiments in a competitive, time-boxed format.
Teams of 3-5 people (cross-functional: product, data, design) compete over a half-day or full day. Each team works through five stages on a shared board: