How to measure AI productivity gains without confirmation bias

Fashion companies are deploying generative AI, demand forecasting algorithms and automated merchandising tools at unprecedented speed, yet many struggle to prove the promised productivity gains. The challenge is not simply measurement but avoiding confirmation bias: the tendency to interpret ambiguous results as evidence that expensive technology investments are working. Without rigorous baselines, control groups or pre-implementation comparisons, executives risk overstating benefits, misallocating capital and making flawed decisions about which AI tools to scale.

The stakes are high. A 2025 McKinsey survey found that while 72 per cent of retail and fashion executives reported adopting at least one AI capability, only 8 per cent achieved material financial impact. The gap reflects a measurement problem as much as an execution one. When productivity gains are claimed without controls, it becomes impossible to distinguish AI's contribution from other factors: improved supplier relationships, better-trained staff, macroeconomic tailwinds or simple regression to the mean after a weak prior period.

This article sets out how fashion businesses can design measurement approaches that withstand scrutiny, address attribution challenges and produce metrics that inform rather than mislead.

Why is confirmation bias so common in AI productivity measurement?

Confirmation bias arises because organisations have strong incentives to validate investments. When a retailer spends millions on an AI-driven assortment planning system, the team that championed the purchase naturally looks for evidence it is working. In the absence of controlled measurement, any improvement in sales, margin or inventory turn can be attributed to the new tool, even if the real driver was currency movements, weather patterns or a competitor's misstep.

The problem is compounded by the complexity of fashion operations. Multiple initiatives often run in parallel: a brand may launch an AI design assistant at the same time it restructures its buying team, renegotiates supplier contracts and opens new distribution channels. Without a methodical way to isolate each variable, the AI tool may receive credit for gains it did not cause, or be blamed for setbacks it did not create.

Vendor-supplied case studies exacerbate the issue. Software providers routinely publish metrics that highlight favourable outcomes without disclosing methodology, sample size or counterfactual scenarios. A claim that a demand forecasting tool improved accuracy by 15 per cent is meaningless without knowing the baseline, the measurement window, whether the comparison was seasonal and whether other process changes occurred simultaneously.

What baseline data should fashion companies establish before deploying AI?

A robust baseline captures performance on key metrics for a defined period before the AI intervention. The baseline period should be long enough to account for seasonality and business cycles,ideally twelve months,and should exclude anomalous events such as pandemic lockdowns or one-off clearance sales. The metrics selected must align with the AI tool's stated purpose: if the tool is intended to speed up design workflows, measure design iterations per week, time from concept to tech pack and designer hours per style. If the goal is inventory optimisation, track sell-through rate, markdown percentage and weeks of supply.

The baseline must also document the broader context. Record staffing levels, average order values, customer acquisition costs, supplier lead times and any other variables that could plausibly influence the target metric. This contextual data allows analysts to control for confounding factors when evaluating post-deployment results.

Critically, the baseline should be locked before deployment. Teams should resist the temptation to revise historical data or cherry-pick favourable comparison periods after seeing initial results. One European luxury group, according to a 2025 Vogue Business report, established a formal protocol requiring baseline approval by finance and operations leaders before any AI pilot could proceed, ensuring that subsequent claims of productivity gains would be auditable.

How do control groups and A/B tests work in fashion operations?

Control groups offer the most rigorous path to isolating AI impact. In a controlled test, one segment of the business uses the AI tool while a comparable segment continues with existing processes. The difference in outcomes between the two groups, measured over the same period, provides a clearer signal of the tool's effect.

Fashion retailers with multiple stores or brands can assign the AI tool to a subset and use the remainder as controls. For example, a chain might deploy an AI-driven replenishment system in half its stores, matched by geography, size and customer demographics, while keeping manual processes in the other half. The comparison reveals whether inventory turn, out-of-stock rates or labour hours improved more in the AI group than in the control group.

A/B testing is feasible for digital functions. An e-commerce team can show AI-generated product descriptions to half of site visitors and human-written copy to the other half, measuring conversion rates, bounce rates and return rates for each cohort. Similarly, merchandisers can test AI-recommended product bundles against human-curated assortments, tracking attach rates and average basket value.

The challenge is maintaining discipline. Control groups must remain truly comparable: if the AI stores also receive additional staff training or new point-of-sale hardware, the test is compromised. Sample sizes must be large enough to detect meaningful differences, and the test period must be long enough to smooth out weekly volatility. A four-week test may suffice for e-commerce experiments; a store-level operational test may require a full season.

What are the main attribution challenges when measuring AI productivity?

Attribution becomes difficult when AI tools are embedded in workflows alongside other technologies and human inputs. A design team may use an AI assistant to generate colour palettes and fabric suggestions, but the final collection still depends on designers' creative judgment, trend research, supplier negotiations and market feedback. If the collection sells well, how much credit belongs to the AI?

One approach is to measure intermediate outputs rather than final business results. Instead of asking whether AI increased total revenue,a metric influenced by countless factors,ask whether it reduced the time designers spent on repetitive tasks, increased the number of concepts explored or improved internal satisfaction scores. These proximal outcomes are more plausibly linked to the AI tool and less confounded by external variables.

Another approach is marginal contribution analysis. If a retailer's gross margin improved by 200 basis points after deploying an AI pricing tool, analysts can model what portion of the gain is attributable to the tool by controlling for changes in product mix, raw material costs, competitor pricing and promotional intensity. Statistical techniques such as regression analysis or difference-in-differences estimation can isolate the AI effect, though they require clean data and statistical expertise.

Time lags complicate attribution further. Productivity gains from AI may not materialise immediately. A demand forecasting algorithm may take several months to learn seasonal patterns and supplier behaviour. A generative design tool may require iterations before designers trust its output and integrate it into their creative process. Measuring too soon risks underestimating impact; waiting too long allows external factors to obscure the signal.

Which metrics matter most for different fashion functions?

Function-specific metrics provide clearer insight than company-wide productivity claims. For design teams, relevant metrics include design cycles (days from brief to approved sample), number of styles developed per designer per season, and cost per prototype. AI tools that automate pattern grading or generate colour variations should demonstrably reduce these timelines or costs.

Merchandising and buying teams should track forecast accuracy (mean absolute percentage error between predicted and actual demand), markdown rates, sell-through at full price and inventory days on hand. AI-driven assortment planning or demand sensing tools should improve these metrics relative to baseline or control groups.

E-commerce functions can measure conversion rates, average order value, cost per acquisition, return rates and customer lifetime value. AI applications in personalisation, search ranking or chatbot support should move these metrics in a favourable direction, measured through A/B tests.

Supply chain and sourcing teams should monitor lead times, on-time delivery rates, defect rates and cost per unit. AI tools for supplier selection, quality prediction or logistics optimisation should reduce lead times or costs without compromising quality.

Customer service and operations teams can track tickets per agent per day, average handling time, first-contact resolution rate and customer satisfaction scores. AI chatbots or agent-assist tools should improve efficiency or quality metrics, ideally both.

Can you share a concrete company example with documented results?

In early 2025, Inditex, the parent company of Zara, disclosed in its annual report that it had deployed AI-driven demand forecasting and inventory allocation tools across its store network. The company reported that the initiative contributed to a reduction in inventory levels relative to sales, with inventory as a percentage of sales declining and stock turn improving compared to the prior year. Inditex attributed part of this improvement to better alignment between store replenishment and real-time demand signals processed by machine learning models.

Importantly, Inditex did not isolate the AI contribution from other operational changes implemented in the same period, including supply chain investments and store format adjustments. The company's disclosure illustrates both the potential of AI tools and the difficulty of clean attribution: multiple initiatives ran in parallel, and the reported metrics reflected the combined effect. Analysts and investors were left to estimate how much of the inventory efficiency stemmed from AI versus broader operational discipline.

The Inditex case underscores the need for rigorous internal measurement even when external reporting remains aggregated. Companies that establish baselines, control for confounding factors and measure intermediate outcomes are better positioned to understand what is working and make informed decisions about scaling or abandoning AI initiatives.

What risks arise from poor AI productivity measurement?

The most immediate risk is misallocation of capital. If executives believe an AI tool is delivering productivity gains when it is not, they may invest further in the wrong technology, expand it to additional functions or purchase similar tools from the same vendor. The opportunity cost is significant: budgets and leadership attention diverted from interventions that might deliver genuine returns.

Poor measurement also creates internal credibility problems. When productivity claims are later revealed to be exaggerated or unsupported, trust in the technology team erodes. Business unit leaders become sceptical of future AI proposals, even those with merit, and the organisation becomes slower to adopt innovations that could genuinely improve performance.

Gaming is another hazard. When teams know their performance will be judged on specific metrics, they may optimise for those metrics in ways that do not align with business goals. A customer service team measured on tickets per agent per day may deflect complex queries to avoid slowing throughput. A merchandising team measured on forecast accuracy may game the metric by inflating safety stock or focusing predictions on high-volume, low-variability items. The measured metric improves, but the underlying business outcome does not.

There are also limits to what measurement can reveal. Productivity gains are only valuable if they translate into business outcomes: higher revenue, lower costs, better customer experience or competitive advantage. A design tool that doubles the number of concepts explored per week is productive only if the additional concepts lead to better-selling collections. A forecasting tool that reduces mean absolute percentage error by 20 per cent is valuable only if the improved accuracy enables better inventory decisions, higher margins or fewer stockouts. Focusing on intermediate metrics without connecting them to business results can create an illusion of progress.

What are the counter-arguments to rigorous AI measurement?

Some executives argue that rigorous measurement protocols slow down innovation. If teams must design control groups, lock baselines and wait months for statistically valid results, the argument goes, the business will fall behind faster-moving competitors who deploy AI tools more aggressively. Speed and experimentation, in this view, matter more than measurement precision.

There is merit to this concern in fast-changing environments. Fashion businesses operate on seasonal cycles, and waiting a full year to evaluate a tool may mean missing the window to scale it for the next season. However, the counter to this counter-argument is that rapid iteration does not preclude measurement. Teams can run shorter, smaller-scale tests and use leading indicators,such as user adoption rates, task completion times or qualitative feedback,to make provisional decisions while still building the infrastructure for rigorous long-term evaluation.

Another objection is that some AI benefits are inherently difficult to quantify. A generative design tool may inspire creativity, improve team morale or help attract talent, even if it does not measurably reduce design cycle time. These intangible benefits are real and may justify the investment, but they should be acknowledged as distinct from productivity gains. Executives should be clear about which benefits are measurable and which are strategic bets based on qualitative judgment.

Finally, some argue that the complexity of modern fashion operations makes clean attribution impossible. If dozens of variables influence a given outcome, isolating the AI effect is a statistical challenge that requires expertise and data infrastructure many companies lack. This is a practical constraint, not a reason to abandon measurement altogether. Even imperfect measurement,tracking a few key metrics with documented baselines and transparent assumptions,is better than relying on anecdote and intuition.

Frequently asked questions

What should fashion companies do now?

1. Establish locked baselines before deployment. Document performance on target metrics for at least twelve months, capturing seasonality and contextual variables. Require sign-off from finance and operations leaders to prevent post-hoc revision.
2. Design control groups wherever feasible. Allocate AI tools to a subset of stores, brands or product categories and maintain comparable segments without the tool. Measure the difference in outcomes to isolate AI impact.
3. Select function-specific metrics. Match measurement to the AI tool's stated purpose: design cycle time for creative tools, forecast accuracy for demand planning, conversion rate for e-commerce personalisation. Avoid vague company-wide productivity claims.
4. Plan for time lags and iterate. Allow six to twelve months for AI tools to mature and for measurable effects to appear. Use leading indicators such as user adoption and task completion to guide interim decisions without abandoning rigorous long-term evaluation.
5. Control for confounding factors. Document all parallel initiatives,hiring, process changes, supplier renegotiations,that could influence target metrics. Use statistical techniques or expert judgment to estimate AI's marginal contribution.
6. Guard against gaming. Monitor whether teams optimise for measured metrics in ways that undermine business goals. Supplement quantitative metrics with qualitative assessment and spot checks of work quality.
7. Connect intermediate metrics to business outcomes. Ensure that improved design cycle time, forecast accuracy or chatbot resolution rates translate into revenue growth, cost reduction or customer satisfaction. Measure the full chain, not just the proximal output.