Most A/B tests are too small to find the effect they're looking for. Working out the sample size before launch is the single most useful habit in conversion optimisation, and it's less mathematical than it looks.
Three inputs
- Baseline conversion rate: how often the step converts today.
- The smallest effect worth detecting: the minimum improvement that would change what you do.
- How sure you want to be: conventionally an 80% chance of detecting a real effect, with a 5% false-positive rate.
A rule of thumb
For those conventional settings, visitors needed per variant are roughly:
16 × p × (1 − p) ÷ d²
where p is the baseline rate and d is the absolute difference you want to detect.
Example: a 3% conversion rate, and you want to detect a 10% relative improvement (3.0% to 3.3%, so d = 0.003). That's 16 × 0.03 × 0.97 ÷ 0.000009, or about 52,000 visitors per variant.
What this tells you
- Small improvements need a lot of traffic. Halve the effect you want to detect and you need four times the visitors.
- Low-converting steps need more traffic than high-converting ones.
- If you can't reach the number in a reasonable time, test a bigger change, test higher in the funnel, or use a different method.
Turning visitors into duration
Divide the visitors needed (for all variants) by the weekly traffic reaching the step, round up to whole weeks, and run for at least one or two full weekly cycles so weekday patterns even out.
Then don't stop early
Decide the duration before launch and read the result at the end. Stopping as soon as the result looks good inflates false positives dramatically. If you need flexibility, use a method designed for it, such as sequential testing, and agree it in advance.
