How to set up control groups for testing AI merchandising tools

What unit should you randomise when testing AI tools?

The choice between randomising individual products, stores or time periods determines both statistical power and the risk of interference between test and control groups. Randomisation is sufficient when the sample size is significantly large, at least 10,000 per group, but insufficient when dealing with small numbers, as is common for retail tests looking at markets or stores, typically fewer than 100.

SKU-level randomisation delivers the highest statistical power for a given sample size because it maximises the number of independent observations. However, this approach only works when products do not compete for shared inventory capacity or customer attention. When testing AI demand forecasting tools, both switchback experiments and item-level randomisation suffer from systematic bias due to interference: temporal carryover in switchbacks and cannibalization across items under capacity constraints.

Store-level randomisation avoids contamination when the intervention affects merchandising decisions, promotional calendars or staff behaviour that cannot be isolated to individual products. Randomization at the store level reduces the risk of selection bias and makes your results more defensible. The trade-off is reduced statistical power, requiring either larger effect sizes or longer test durations to reach significance.

How do you calculate the required sample size?

Sample size depends on three parameters: the baseline conversion or sell-through rate, the minimum detectable effect you want to measure and the statistical power threshold. Convention allows 80% to be a sufficient level of power. Typical controlled trials set the statistical significance level at 0.05 or 0.01 and the power at 80 or 90 percent.

The relationship between these parameters is non-linear. A 50% drop in effect size will require a four-fold increase in sample size to achieve the same power. For e-commerce tests, a store converting 2.0% of sessions, testing a change that should lift conversion to 2.2%, a 10% relative lift, requires several thousand sessions per variant to detect the difference reliably.

When testing AI markdown tools at store level, the calculation must account for clustering effects. If it is a cluster RCT, the intra-cluster correlation reduces effective sample size because stores within the same region or format tend to behave more similarly than stores selected at random. Publicly available sample size calculators for two proportions provide the baseline estimate, which should then be adjusted upward based on the expected correlation within clusters.

How should you match control stores to test stores?

When sample sizes fall below 10,000 per group, intelligent allocation into matched groups ensures balance on variables that drive the outcome metric. When the sample size is small, we intelligently allocate each subject into testing and control groups so that all the groups look alike as much as possible.

Store clustering provides the foundation for matching. Retailers cluster stores by grouping those with similar customer demand, then plan assortments, size allocations and promotions for each cluster. Common bases are store capacity, store attributes and demographics, category sales, productivity and price. The variables used for clustering should reflect the mechanics of the AI tool being tested. When evaluating demand forecasting software, cluster on historical sales volatility, rate of new product introduction and channel mix. When testing markdown optimisation, cluster on price sensitivity, promotional frequency and end-of-season sell-through patterns.

When clusters of individuals are randomized rather than the individuals themselves, the best practice should involve three steps. Researchers should randomly choose one treated and one control cluster within each pair. This matched-pair design protects against imbalance when the total number of stores is small, typically below 50 locations.

What types of bias should you control for?

Selection bias arises when assignment to test or control depends on characteristics that also influence the outcome. Self-selection bias can make it challenging to accurately evaluate certain types of initiatives, such as loyalty programs. If merchandisers volunteer their categories for an AI forecasting pilot, those categories may already have cleaner data or more engaged teams, inflating the observed effect.

Random assignment is the primary defence against selection bias, but only when the randomisation is enforced before any intervention begins. Using historical performance as a proxy for a control group is the most common shortcut in retail experimentation and one of the most problematic. Comparing test period performance against the same period last year feels intuitive, but last year is not a control group. A contemporaneous control group, stores running in parallel during the same test period, is always more reliable than a historical comparison.

When true randomisation is not feasible, synthetic control methods can reduce bias in retrospective analyses. The provider used Test & Learn to build synthetic control groups based on customer profiles. Synthetic control methodology is the process of retroactively identifying well-matched control groups in situations where there was no initial randomized control group. However, these methods require strong assumptions about which variables drive outcomes and cannot fully substitute for prospective randomisation.

Attrition bias occurs when participants drop out of the test non-randomly. In store-level tests, this might mean stockouts that prevent a markdown tool from executing its recommended prices, or system integration failures that exclude certain locations from receiving AI-generated forecasts. Non-differential attrition occurs when attrition from the treatment or control groups occurs for reasons unrelated to the treatment. In this case, we would only worry if non-differential attrition erodes our sample size such that issues of statistical significance or power crop up.

How long should the test run?

Test duration must satisfy two constraints: accumulating the pre-calculated sample size and capturing at least one complete business cycle. At minimum, run every test for one full business cycle (7 days) to account for day-of-week variations, even if you reach your target sample size sooner. For AI tools that influence buying decisions or markdown timing, one business cycle means one full season, from initial allocation through end-of-season clearance.

Stopping a test early when results appear significant increases the false positive rate. Peeking too early: Checking results before your sample is sufficient can inflate Type I error (false positives). Set the sample size and end date before launching the test, and resist pressure to call a winner based on interim results unless the test design explicitly accounts for sequential analysis.

Take-up rates can sometimes be lower than expected, and this can have consequences on your effect size and, following that, on your statistical power. If operational issues prevent the AI tool from activating in some test locations, the effective treatment intensity decreases, requiring either a longer test window or acceptance of a higher minimum detectable effect.

What operational controls protect validity?

Document the test protocol before randomisation, including the hypothesis, the primary outcome metric, secondary metrics and the planned analysis approach. This pre-registration prevents selective reporting of whichever metric shows the most favourable result.

Ensure that the intervention is the only systematic difference between test and control groups. When testing an AI demand forecasting tool, both groups should use the same ERP system, the same buying calendar and the same approval thresholds for purchase orders. The only variable should be whether the forecast comes from the AI model or the incumbent process.

Monitor concurrent experiments that might interact with the AI tool test. If the marketing team launches a promotional test in the same stores during the same window, attribution becomes ambiguous. Coordinate the test calendar across merchandising, marketing and operations teams to maintain clean treatment assignment.

Verify that data collection mechanisms work identically in test and control groups. If the AI tool writes its recommendations to a dashboard that buyers check daily, but the control group relies on weekly email reports, observed differences might reflect information accessibility rather than forecast accuracy. Measurement infrastructure should be identical across both arms.