<aside> ☝🏽
Most organizations start experimentation the same way: one team, one tool, a few passionate people running tests. It works for a while. Then demand increases, more teams want to test, and the central team becomes the bottleneck. Every hypothesis goes through the same 3 people. Turnaround slows down. Teams get frustrated and either stop testing or start running tests without oversight (which is worse).
This is the moment where the organizational model matters more than the tool.
</aside>
There are three ways to structure an experimentation function. Each has clear trade-offs.
1. Centralized model. One team owns all experimentation. They design tests, configure them, analyze results, and present conclusions. The upside: quality control is easy, methodology is consistent. The downside: it doesn't scale. A team of 5 cannot support 60 product teams. You become a service desk, not a capability builder.
2. Fully decentralized model. Every product team runs their own experiments. The upside: speed and ownership. The downside: without governance, quality collapses. I've seen organizations where 78% of decentralized experiments had methodology flaws, over half were duplicates of previous tests, and only 11% were properly implemented after showing positive results.
3. Center of Excellence (CoE) with distributed champions. A small central team owns strategy, methodology, tooling, and governance. Product teams own execution: they design and run their own experiments, supported by local champions. The central team sets the rules, the distributed teams play the game.
This third model is what I call disciplined decentralization. It's the only model I've seen work at scale.
The CoE is not a testing team. It's an enablement team. The shift in identity matters. You stop being the people who run tests and start being the people who make testing possible for everyone else.
Methodology and standards. What constitutes a valid hypothesis, what guardrails are mandatory, what statistical thresholds apply, how results are documented. These are not suggestions. They are enforced through tooling and process gates (see 🔄 The experimentation process).
Tooling and infrastructure. The experimentation platform, the power calculator, the results dashboard, the experiment repository.
Governance and quality. Monitoring process compliance, running quarterly retros on experimentation quality, flagging zombie tests, ensuring the "no significance, no go-live" policy is respected.
Training and onboarding. Building the learning center, running onboarding cohorts for new teams, maintaining documentation, organizing hackathons.
AI-assisted tools. The Hypothesis Coach, Testing Compass, and Related Tests agent (see 🤖 AI tools for experimentation). These scale the central team's expertise without scaling their headcount. AI tools for experimentation
Hypothesis generation. They know their product, their users, and their data better than anyone.
Test execution. Setting up experiments, running QA, monitoring guardrails during the test.
Decision-making. The team decides whether to ship based on results, within the governance framework.
Documentation. Writing the conclusion, capturing learnings in the repository. Non-negotiable, enforced as a gate before any new test can start.