<aside> ☝🏽
Standard A/B testing assumes you can randomly assign individual users to treatment and control. For most product experiments, this works. But roughly 30% of the business questions I encounter cannot be answered this way.
</aside>
Pricing is the classic example. When prices are set at the product level, the route level, or the market level, you cannot show different prices to different users on the same product without creating a legal, ethical, or operational mess. Supply-side changes, marketplace interventions, algorithm updates that affect all users equally, operational policy changes: none of these can be randomized at the user level.
The usual response is to skip testing entirely and ship based on judgment. That is a missed opportunity. There are rigorous experimental and quasi-experimental methods designed specifically for these situations. They require more statistical sophistication than a standard A/B test, but they produce actionable causal evidence where the alternative is no evidence at all.
Switchback experiments are the method I use most for pricing and other non-randomizable tests. The core idea is simple: instead of randomizing across users, you randomize across time periods and clusters.
Each cluster (a route, a market, a product category) alternates between treatment and control over time. Route A gets the treatment price in week 1, the control price in week 2, back to treatment in week 3, and so on. Each cluster serves as its own control, which eliminates the confounding from permanent differences between clusters.
This design was validated at scale by LATAM Airlines for ancillary pricing experiments (Olivares et al., 2024) and is used across industries where user-level randomization is not feasible.
The design has five key components:
1. Define experimental clusters. Identify the units you will randomize over. For pricing, this is typically a route, a market, or a product category. The critical requirement: clusters must be independent. A price change on Cluster A must not mechanically affect demand on Cluster B. If customers can easily substitute between clusters, independence is violated and results are biased.
2. Build balanced pods. Group similar clusters into pods using k-means clustering on key variables: average revenue, demand volatility, seasonality patterns, market characteristics. Within each pod, half the clusters start on treatment and half on control. This ensures that at any given time, your treatment and control groups are structurally balanced. Without this step, you risk all high-revenue clusters landing in treatment by chance.
3. Define switching cadence. Decide how often clusters flip between treatment and control. Weekly switching is the sweet spot for most applications: it provides enough data per period while aligning with natural business cycles. Daily switching gives more precision but requires operational capability to switch reliably every day. Bi-weekly is an option when effects take longer to stabilize.
4. Include washout periods. At each switch, exclude 1-2 days of data from analysis. When you change a price or policy, there is a transition period before the new regime reaches full effect. Booking windows, caching, delayed behavioral responses: all of these mean the first day after a switch is noisy. The washout period removes this noise.
5. Lock the schedule before launch. Pre-register the full switchback schedule. Which cluster gets treatment or control on which dates. No modifications once the experiment starts. This is the equivalent of pre-registration in standard A/B testing and prevents the temptation to adjust mid-experiment.
The analysis uses Two-Way Fixed Effects (TWFE) regression:
Y(i,t) = route_fixed_effect + time_fixed_effect + beta x Treatment(i,t) + error
The route fixed effect absorbs everything permanently different between clusters (Route A always earns more than Route B). The time fixed effect absorbs everything that moves all clusters on a given day (holidays, seasonality, macro events). What remains is the treatment effect: the causal impact of your intervention, isolated from both cluster differences and temporal shocks.