Skip to content
home

Conversion optimisation · 2 min read

A/B/n tests: why more variants need more traffic

Each extra variant needs its own sample and raises the risk of false wins. Here's how to plan for it.

By WebIQ Analytics · 6 August 2026

Testing three or four versions at once looks efficient: one test instead of several. In practice, every extra variant needs its own share of traffic, and comparing many variants with control raises the chance of a false winner. Both make A/B/n tests slower than they look.

More variants split the traffic

Each variant needs enough visitors to detect the effect you care about. With a 3% baseline conversion rate and a 10% relative improvement to detect, a standard A/B test needs about 53,000 visitors per variant, or roughly 106,000 in total.

More comparisons, more false wins

Compare three variants with control at the usual 5% significance level and the chance that at least one looks like a winner by luck rises well above 5%. A common fix is to divide the significance level by the number of comparisons (the Bonferroni correction). It's simple and conservative, and it means each variant needs more visitors.

Total visitors needed as the number of variants grows.106,415A/B (2)193,3043 variants283,8804 variants377,9675 variantsTotal visitors neededExample: 3% baseline, 10% relative lift, 80% power, α 5% with Bonferroni correction.
Total visitors needed as the number of variants grows.

With that correction, total traffic for the same example grows to about 193,000 visitors with three variants, 284,000 with four and 378,000 with five.

When A/B/n makes sense

  • High-traffic pages, where the extra sample is affordable.
  • Genuinely different approaches to the same problem, where you want to compare directions, not tweaks.
  • Early exploration, followed by an A/B test of the winner against control.

When it doesn't

  • Low-traffic pages, where even one variant struggles to reach its sample.
  • Small variations of one idea, where differences between them will be tiny.

Plan it properly

  1. Decide the number of variants before launch.
  2. Size the test with the correction for multiple comparisons.
  3. Allocate traffic equally unless there's a reason not to.
  4. Read out once, at the planned sample, comparing each variant with control.
  5. Confirm a winner with a follow-up A/B test if the result is close.

Fewer, well-chosen variants usually reach a clear answer faster than many variants that never reach their sample.

Experiments that tell you the truth about your site

We run research-led testing programmes on your site and funnels, designed to detect real effects and read out without wishful thinking.