How to set up a control group for testing AI merchandising tools

Testing AI merchandising tools demands a structured control group design to separate the technology's effect from seasonal trends, promotional activity, and external shocks. A rigorous A/B test compares stores or product categories that receive AI-driven assortment and inventory recommendations against a matched control group that continues business as usual, allowing retailers to measure incremental performance with statistical confidence.

Why is randomisation critical for valid results?

Random assignment to treatment and control groups is the foundation of causal inference, ensuring that any performance difference can be attributed to the AI tool rather than pre-existing store characteristics. However, pure randomisation may place high-volume flagship stores in one group and smaller outlets in another, introducing bias. Stratified randomisation solves this by first grouping stores into strata based on sales volume, geographic region, customer demographics, and store format, then randomly assigning stores within each stratum to treatment or control. This preserves balance on observable characteristics that drive sales.

Paired matching offers an alternative when sample sizes are small. Each treatment store is paired with a control store that closely resembles it on historical sales, foot traffic, and assortment depth. This approach increases statistical power but requires careful documentation of matching criteria and sensitivity analysis to test whether results hold under different pairing schemes.

Which baseline metrics should you track?

Establishing baseline performance during a pre-test period, typically four to eight weeks before the AI tool goes live, allows you to verify that treatment and control groups are comparable and provides a reference point for measuring change. Key metrics for merchandising tests include sell-through rate, inventory turnover, stock-out frequency, markdown rate, and gross margin. Collecting these metrics at the store-week or SKU-week level enables granular analysis and helps identify whether the AI tool improves performance uniformly or only for certain product categories or price points.

During the baseline period, check that the two groups do not differ significantly on any metric using t-tests or non-parametric equivalents. Material imbalances suggest that randomisation failed or that stratification was incomplete, requiring either re-randomisation or statistical adjustments such as regression controls or propensity score weighting.

How large should your sample be?

Sample size determines whether the test can detect a meaningful effect with acceptable statistical power, typically 80 per cent. Power analysis requires three inputs: the minimum detectable effect size, the baseline metric's standard deviation, and the desired significance level, usually 5 per cent. For example, if baseline sell-through is 65 per cent with a standard deviation of 10 percentage points, detecting a 3 percentage point improvement requires approximately 175 stores per group. Smaller effect sizes or noisier metrics demand larger samples.

Retailers with fewer stores can increase power by testing at the product category or SKU level rather than the store level, or by extending the test duration to accumulate more observations. However, category-level tests raise the risk of spillover effects if the AI tool reallocates inventory across categories, contaminating the control group. Store-level tests avoid this issue but require sufficient geographic or operational separation to prevent cross-contamination.

What test duration captures real performance?

AI merchandising tools optimise inventory allocation and replenishment over weekly or bi-weekly cycles, so tests must run long enough to observe multiple decision cycles and account for demand variability. A minimum duration of 12 weeks is standard, though full-season tests, spanning 16 to 26 weeks, better capture the AI tool's ability to respond to shifts in customer preferences, weather patterns, and promotional calendars. Shorter tests risk false positives if they coincide with temporary demand spikes or operational disruptions that affect treatment and control groups differently.

Seasonal merchandise poses a particular challenge: testing a tool during peak selling weeks may overstate benefits if the AI simply rides strong demand, while off-peak tests may understate performance gains. Staggered rollouts, where different store cohorts receive the AI tool at different times, can control for seasonality and provide multiple independent estimates of the treatment effect.

How do you analyse and validate the results?

After the test concludes, compare treatment and control groups on each primary metric using difference-in-differences analysis, which nets out any baseline imbalance and common time trends. For example, if sell-through in the treatment group rose from 65 to 70 per cent while the control group increased from 65 to 67 per cent, the treatment effect is 3 percentage points. Standard errors should account for clustering at the store level if individual SKUs or weeks are the unit of observation, as performance within a store is likely correlated over time.

Statistical significance does not guarantee business relevance. A 1 percentage point improvement in sell-through may be statistically detectable in a large sample but economically trivial relative to the AI tool's cost. Conversely, a large point estimate with wide confidence intervals may reflect genuine benefits but insufficient sample size. Sensitivity checks, such as excluding outlier stores or adjusting for concurrent promotions, test whether the main result is robust or driven by a few unusual observations.

What are common pitfalls to avoid?

Several design flaws undermine A/B tests of merchandising tools. Testing only in high-performing or low-performing stores biases the sample and limits generalisability. Allowing store managers to opt in or out of the treatment group introduces selection bias, as managers who volunteer may differ systematically in skill or motivation. Changing the AI tool's configuration mid-test, or rolling it out to additional stores before the scheduled end date, breaks the experimental design and confounds the treatment effect with time-varying factors.

Data quality issues also pose risks. Missing or inconsistent POS data, delayed inventory updates, and manual overrides of AI recommendations all introduce measurement error that attenuates estimated effects. Establishing data validation protocols and tracking recommendation adherence rates ensures that the test measures the AI tool's true impact rather than implementation failures.

• Define success metrics and minimum detectable effects before the test begins to avoid selective reporting of favourable outcomes.
• Document all design decisions, including stratification variables, matching algorithms, and sample exclusions, to enable replication and audit.
• Plan for null results by identifying operational improvements or configuration changes to test in a follow-up experiment.
• Communicate results transparently, reporting confidence intervals and effect sizes alongside p-values to give stakeholders a complete picture of uncertainty.