<aside> ❓
"How mature is our experimentation program?" is one of the most common questions I get from leadership. It's also one of the most dangerous, because the standard answers are almost always wrong.
</aside>
Let me start with what maturity is not.
Myth 1: "We have a big team, so we're mature." One e-commerce company had 200 tests per year, 15+ specialists, and only 15% of experiments guided strategic decisions. Team size says nothing about impact.
Myth 2: "We run a lot of tests." Test velocity is an activity metric, not an outcome metric. Running 400 tests annually is not maturity if most are low-impact micro-optimizations that don't connect to strategic goals. A fintech startup with 2 part-time practitioners had higher maturity because every test was strategically aligned.
Myth 3: "We have sophisticated tools." Having tools does not equal being data-driven. I've seen organizations with best-in-class platforms where the tools collect dust because nobody trusts the results or acts on them. A "Frankenstack" of disconnected tools (Jira for planning, spreadsheets for tracking, Notion for docs, a separate analytics platform) creates the illusion of process while preventing actual governance.
Myth 4: "We democratized access." Self-service experimentation without governance is chaos. Siloed learning, leadership blindness, quality degradation. Democratization without enforcement is the fastest path to an untrusted program.
<aside> ☝🏽
Key takeaway: True maturity is not about volume, team size, or tools. It's about the percentage of experiments that actually influence strategic decisions.
</aside>
Real maturity has a simple test: Do executives cite specific experiments that changed their mind on important decisions?
If the answer is yes, you're mature. If the answer is "they attend the review meeting" or "they like the dashboard," you're not there yet. Passive attendance is not buy-in.
I think of maturity across four levers:
1. Decision-making & leadership. Do experiment results actually drive shipping decisions? Is there a "no significance, no go-live" policy? Does leadership ask "what did we test?" before signing off on features?
2. Process. Is the experimentation process enforced, not just documented? Are hypotheses pre-registered? Are results documented? See 🔄 The experimentation process.
3. Infrastructure & tools. Is there a unified platform? Does it support guardrails, automated notifications, and quality gates? Can teams self-serve without sacrificing governance?
4. Experiment setup & management. Do teams know how to design proper tests? Are champions in place? See 🏛️ The Center of Excellence model.
To track maturity across a complex organization with many product teams, I use a visual maturity mapping exercise. Think of it as a radar chart with your product areas on the spokes and maturity levels on the rings:
Inner ring: Accountable. The team has acknowledged experimentation as part of their process.
Second ring: Active. The team has run experiments and is developing the habit.