How to test a change that only affects discounted products

Last week we discussed a question that looked simple: how do you test a change that only affects discounted products? We were improving how product cards communicate discounts, with better labels and clearer pricing. We wanted to measure product clicks and add-to-cart, but only for discounted products, and those make up about 10-15% of the assortment at any given time.

The first idea was to run the test on everyone and evaluate only the people who came into contact with discounted products. It looked clean, because we would compare the same kind of people in variant A and B. But product cards are everywhere, on the homepage, in categories and in search, so the change can affect how people see the whole site.

So I objected: what if the new labels reduce interest in full-price products? Interest in discounts would go up, but total revenue would drop. The labels can also change how people perceive product value or teach them to wait for sales. If we evaluated only the people who clicked a discounted product, we would see none of this. We would miss customers who would have bought at full price but got distracted by a discount, the shift in purchase mix and the impact on revenue and margin.

What we decided

We run the test on everyone, and revenue is the primary metric. We analyze interest in discounted products only after we confirm the change doesn’t hurt revenue.

The price is a longer test. The effect spreads across all users, not only those who see discounts, and a smaller overall effect needs more data before we can confirm it with confidence. In return we see the impact on the whole business, and we don’t risk improving one segment while overall performance gets worse.

When to filter

The discussion left us with one question we now ask for a test: can the change indirectly affect behavior outside the place where it is visible? The size of the affected segment doesn’t decide it.

Discount labels are a visual change across the whole site, so they can shift attention and purchases between products. That’s why we went with the full sample. The same applies to a header or navigation redesign and to changes where the business cares about shifts between categories.

For an isolated feature, filtering makes sense. If 5% of people use search, we evaluate a search algorithm change on them, and not on the 95% who never use it. A product detail page redesign only needs the 60% of people who reach a product page, and a checkout change only the 15% who get to checkout.

Filtering is only safe if you log exposure in both variants. In the control group, record the people who would have seen the change too, otherwise the groups aren’t comparable. This approach is called triggered analysis.

When we aren’t sure, we start with the full sample and the primary metrics. We can run a filtered analysis later, but we can’t get back data on effects we never measured. We’re still fine-tuning this, so if you’ve dealt with a similar dilemma, write to me and tell me how you approached it.