<aside> 👀

AI in experimentation is not about replacing statisticians or automating decisions. It's about reducing cognitive load for the people who actually run tests day to day: product owners, business analysts, and experimentation champions.

</aside>

In a large enterprise running hundreds of experiments per year, the bottleneck is rarely the testing tool itself. It's the human steps before and after: framing a good hypothesis, choosing the right validation method, and finding related tests from the past. This is where AI creates the most value.

I think of it this way: AI didn't create our capability. It activated our memory.

The testing compass: should you even run an A/B test?

One of the most common mistakes teams make is jumping straight to an A/B test when a simpler validation method would be faster and cheaper. Not every question needs a controlled experiment to get an answer.

The Testing Compass is a decision tree that guides product owners and business analysts to the right methodology based on where they are in their project. It works as a gatekeeper: it prevents teams from spending weeks on an expensive A/B test without proper foundational validation.

The logic is a sequence of 7 yes/no questions:

Q1: "Do I understand perfectly the user problem?"

If no → Do user research first (interviews, analytics, surveys, journey mapping). You cannot test a solution to a problem you don't understand.

Q2: "Do I know what solution I want to test?"

If no → Do low-fidelity prototyping. Create 2-3 wireframes, test with 5-8 users, gather feedback before building anything.

Q3: "Is this solution particularly expensive or slow to build, with unclear demand?"

If yes → Run a fake door test. Build a landing page with the value proposition, add a signup form, drive traffic. If conversion is above 15%, proceed to build. This saves months of development.

Q4: "Do you have a well-formed hypothesis?"

If no → Stop. Go back and formulate a proper hypothesis before proceeding (see the hypothesis evaluation framework below). Running a test without a clear hypothesis is a waste of traffic.

Q5: "Does this metric really mean anything, and can we measure it?"

If no → Stop. A test without a measurable, meaningful metric cannot produce actionable results.

Q6: "Is there any possible outcome that will change the course of our actions?"

If no → Don't test. If you're going to ship the feature regardless of results, you're not experimenting. You're performing theater.