<aside> ☝🏽
Not all experiments ask the same question. And if the question is different, the statistical test should be different too.
</aside>
Most experimentation platforms default to a two-tailed test. That is the right default for most situations. But there is a category of experiments where a two-tailed test wastes time and traffic, and where a one-tailed test is not only acceptable but methodologically correct.
The distinction starts with the hypothesis.
Every experiment falls into one of three categories based on what you are trying to learn. The category determines the test type, the analysis approach, and the decision framework.
"We believe that changing X will improve metric Y."
This is the classic A/B test. You have a new design, a new feature, a new flow. You want to know: does it perform better than the current version? You care about both directions. Maybe it improves conversion. Maybe it hurts it. Maybe there is no difference. All three outcomes are informative.
Test type: Two-tailed. The significance threshold is split between both tails of the distribution (typically alpha = 0.05, so 0.025 per tail).
When to use: Most product experiments. New features, redesigns, copy changes, funnel optimizations. Any time you genuinely want to know whether the change helps, hurts, or does nothing.
Sample size: Largest of the three types because the significance threshold is split across both directions.
"We believe that changing X will not negatively impact metric Y."
This is fundamentally different from optimization. You are not trying to improve anything. You are shipping a change for other reasons (technical migration, compliance, UX consistency, cost reduction) and you need to verify it does not cause harm.
Consider a prominent login redesign. The team needs to ship it for security reasons. Nobody expects it to increase conversion. The only question is: does it hurt conversion? If it does not, ship it. If it does, investigate.
Test type: One-tailed. The entire significance threshold is concentrated in one direction (alpha = 0.05, all in the negative tail). This means you need roughly half the sample size compared to a two-tailed test for the same power to detect a negative effect.
When to use: Technical migrations, infrastructure changes, compliance updates, mandatory UX changes, platform consolidations. Any change where the business decision is already made and the experiment exists to detect harm, not to justify the change.
Sample size: Smaller than two-tailed for the same power, because all of your statistical budget is focused on the direction you care about.