Path: Foundation (all roles) | Time: 12 min

<aside> 💡

TL;DR: Every experiment you document makes the next experiment better for someone you will never meet. Not just because they might read it, but because a system reads all of them and synthesizes patterns no individual could see across hundreds of experiments. Documentation is not admin. It is how experimentation compounds.

</aside>

Why documentation is the most skipped and most valuable phase

I have audited dozens of experimentation programs. The pattern is the same everywhere. Teams run experiments. They get results. They make decisions. Then they move on without writing down what they learned.

Six months later, a different team runs the exact same experiment on the exact same surface. They get the exact same result. They have learned nothing from the first team's work because it was never captured.

At 50 experiments per year, this is annoying. At 500 experiments per year, it is catastrophic.

What goes into an experiment document

Every write-up captures five things:

1. The hypothesis (verbatim). The exact hypothesis as written before the test started. No retroactive editing.

2. The design. Experiment type, audience, traffic allocation, duration, primary metric, guardrail metrics, sample size calculation.

3. The results. Point estimates, confidence intervals, p-values. Health check outcomes. Anomalies observed during monitoring.

4. The decision. Ship, iterate, or kill, and why. If the decision deviated from the pre-registered protocol, explain the reasoning.

5. The learning. Not "conversion went up by 2%." That is a result. A learning is: "Users respond more to visual progress indicators than to text-based ones. This suggests our checkout flow is perceived as unpredictable, and reducing uncertainty increases completion." A learning is transferable. It applies beyond this specific experiment.

The experiment repository

Documenting experiments in individual pages or Slack threads is documentation theatre. Nobody will search 500 individual pages.

A proper repository is organized by strategic questions, not by team or date:

• "How do users respond to pricing transparency?"
• "What drives first-time booking completion?"
• "Does personalization increase engagement or overwhelm users?"

Experiments are tagged with the strategic questions they inform. When a new team wants to test pricing transparency, they browse all prior experiments on that topic, across all teams, across all time periods.