The problem with simple attribution
The most common way to measure recommendation performance is to count the revenue from shoppers who clicked the widget. It is convenient and wrong: among those shoppers were people who would have bought anyway. The real added value is only the incremental part.
Important: attributed revenue with no control group is an upper bound on the effect, not the effect. The gap can be substantial — particularly in popular categories, where the shopper would have found the product regardless.
Attribution methods in e-commerce personalization
Click-based attribution
The simplest: credit all revenue from shoppers who clicked a recommendation within the attribution window. It overstates the effect. Useful for operational monitoring, not for strategic decisions.
A/B attribution
The correct method: compare two groups, with and without the recommendation, and treat the revenue difference as the incremental effect. It requires a configured test, and it gives a precise answer.
Group A (no recommendation): RPV = 4.80
Group B (with recommendation): RPV = 5.28
Incremental effect: +0.48 per visitor (+10%)
Holdout attribution
The long-run variant: a group of users is excluded from personalization for three to six months. The difference in total revenue shows the cumulative effect of the whole programme rather than of one widget.
Attribution window: how many days?
| Initiative type | Recommended window |
|---|---|
| Onsite recommendations | 7–14 days |
| Popup or banner | 1–7 days |
| Email with recommendations | 7–14 days |
| Push notification | 1–3 days |
The window choice moves the final number: 30 days inflates attributed revenue relative to 7. What matters most is using the same window when comparing strategies.
Cannibalisation: the hidden threat
Cannibalisation is when a recommendation redistributes purchases inside a session without raising total revenue. The shopper intended to buy product A, saw a recommendation for B and bought B instead. The click is credited, and total revenue did not move.
An A/B test at the level of group revenue — revenue per visitor rather than revenue from clicks — neutralises this automatically, because what was bought does not matter, only the total per group.