Skip to content
home

Conversion optimisation · 2 min read

A/B test sample size without the maths anxiety

How big a test needs to be, explained with one chart and one rule of thumb.

By WebIQ Analytics · 4 December 2025

Most A/B tests are too small to find the effect they're looking for. Working out the sample size before launch is the single most useful habit in conversion optimisation, and it's less mathematical than it looks.

Three inputs

  1. Baseline conversion rate: how often the step converts today.
  2. The smallest effect worth detecting: the minimum improvement that would change what you do.
  3. How sure you want to be: conventionally an 80% chance of detecting a real effect, with a 5% false-positive rate.

A rule of thumb

For those conventional settings, visitors needed per variant are roughly:

16 × p × (1 − p) ÷ d²

where p is the baseline rate and d is the absolute difference you want to detect.

Example: a 3% conversion rate, and you want to detect a 10% relative improvement (3.0% to 3.3%, so d = 0.003). That's 16 × 0.03 × 0.97 ÷ 0.000009, or about 52,000 visitors per variant.

Illustrative: the smallest effect a test can reliably detect falls as it runs longer.0%2%4%6%8%10%4w6w8w10w12w14w16w18w20wTest duration (weeks)Minimum detectable effect (80% power)Planned: 12 weeks
The smallest effect a test can reliably detect falls as it runs longer.

What this tells you

  • Small improvements need a lot of traffic. Halve the effect you want to detect and you need four times the visitors.
  • Low-converting steps need more traffic than high-converting ones.
  • If you can't reach the number in a reasonable time, test a bigger change, test higher in the funnel, or use a different method.

Turning visitors into duration

Divide the visitors needed (for all variants) by the weekly traffic reaching the step, round up to whole weeks, and run for at least one or two full weekly cycles so weekday patterns even out.

Then don't stop early

Decide the duration before launch and read the result at the end. Stopping as soon as the result looks good inflates false positives dramatically. If you need flexibility, use a method designed for it, such as sequential testing, and agree it in advance.

Experiments that tell you the truth about your site

We run research-led testing programmes on your site and funnels, designed to detect real effects and read out without wishful thinking.