Path: Foundation (all roles) | Time: 8 min
<aside> 💡
TL;DR: Most product ideas fail. The data is unambiguous: between 70% and 90% of features shipped by experienced product teams do not improve the metrics they were designed to move. Experimentation is how you find the 10-30% that actually work, before committing resources to full rollout.
</aside>
I want to start with a number that changed how I think about building products. Ron Kohavi, who ran experimentation at Microsoft and Bing for over a decade, published this finding: across thousands of A/B tests at Microsoft, only about one-third of ideas that seemed promising actually improved the target metric. At Bing specifically, only 10-20% of experiments showed statistically significant positive results.
This is not a Microsoft problem. Netflix, Google, Booking.com, and Amazon report similar ratios. The pattern is consistent: most ideas that smart, experienced product people believe will work, do not.
Sit with that for a moment. If you are shipping features without testing them, you are statistically likely to be making the product worse more often than you are making it better. Not because you are bad at your job, but because product development is genuinely hard, user behavior is genuinely unpredictable, and intuition is a genuinely unreliable guide at scale.
Experimentation is a method for making product decisions based on evidence rather than opinion. The core mechanic is simple: you show one version of something to a randomly selected group of users (the treatment) and the current version to another group (the control). You measure what happens. You compare.
If the treatment performs better on the metrics you care about and does not degrade the metrics you are protecting, you ship it. If it does not, you do not. The decision is based on data, not on who has the strongest opinion in the room.
This sounds straightforward, but the implications are profound. It means:
The HiPPO loses its veto. HiPPO stands for Highest Paid Person's Opinion. In organizations without experimentation, product decisions are often made by the most senior person in the room. Experimentation replaces authority-based decisions with evidence-based decisions. A junior PM's idea that wins the test beats the VP's idea that does not.
"Failure" becomes learning. When an experiment shows no significant difference or a negative result, that is not a failure. That is a validated learning: you now know that this idea does not work for these users in this context. That knowledge is worth more than an untested launch, because it prevents you from investing further in the wrong direction.
Velocity increases, not decreases. A common objection is that experimentation slows you down. The opposite is true at scale. Without experimentation, you ship everything, then spend months trying to figure out what worked and what did not. With experimentation, you learn in 2-4 weeks, kill what does not work, and double down on what does. Your portfolio of shipped features improves because you stop accumulating dead weight.
Organizations that take experimentation seriously change how they talk about results. This is not cosmetic. Language shapes thinking.
Instead of "the test succeeded" or "the test failed," we say: validated, not validated, or no significant difference.
A validated result means the data supports the hypothesis with statistical confidence. A not-validated result means the data contradicts the hypothesis. No significant difference means we could not detect an effect, which could mean there is no effect, or that our test was not powerful enough to detect a small one.
This vocabulary matters because it removes the emotional charge from results. Nobody "failed." The hypothesis was not validated. The team learned something. They move on to the next experiment with better information.
At scale, experimentation transforms how product organizations operate. Teams that run 50+ experiments per year develop a different relationship with risk, with data, and with each other.