The two factors behind duration
Duration is set independently by two parameters, and the real duration is the larger of the two.
1. The statistical factor — how many users the calculation requires, which depends on the MDE, the significance level and the power.
2. The temporal factor — at least two full weeks regardless of traffic. That covers at least two complete business cycles (weekdays plus weekends, twice).
Traffic: 20,000 unique visitors per week
Baseline CR: 2.5%
MDE: 10% relative → 2.75%
Required sample: ~65,000 per variation
Computed duration: 65,000 / 10,000 = 6.5 weeks
Minimum duration: 2 weeks
Result: 6.5 weeks, round to 7
Why whole weeks matter
Shopper behaviour differs substantially by day of week. Weekends often convert higher in some categories and lower in others. A test started on a Wednesday and stopped the following Monday contains two weekends in one group and one in the other — a structural bias in the sample.
Rule: start a test early in the week (Monday or Tuesday) and schedule the stop at the same time N whole weeks later.
Duration and the bandit
A multi-armed bandit has no fixed duration in the classical sense — traffic shifts dynamically toward the better variation. Even so, a minimum period of 7–14 days before aggressive reallocation is advisable, to avoid the novelty effect — the short-lived metric lift any new variant enjoys.
Duration at low traffic
| Weekly traffic per variation | Realistic duration (MDE 10%) |
|---|---|
| 5,000 | 10–12 weeks |
| 10,000 | 6–7 weeks |
| 25,000 | 3–4 weeks |
| 50,000+ | 2 weeks |
Below 3,000–5,000 unique visitors a week, classical A/B testing loses its meaning — the test would take months. In that situation, ship the change on expert judgement and follow it with cohort analysis.