Skip to content
home

Conversion optimisation · 2 min read

Why most A/B tests can't detect the effect they're looking for

Underpowered tests, peeking and what to do when traffic is limited.

By WebIQ Analytics · 8 January 2026

A common pattern: a team runs a test for two weeks, sees no significant result, concludes "it made no difference", and moves on. Often the test never had a realistic chance of detecting the effect.

Underpowered by design

If a page converts at 2% and the test aims to spot a 5% relative improvement, it needs hundreds of thousands of visitors per variant. Run it on a fraction of that and "no significant difference" means "we couldn't tell", not "there's no effect".

Peeking

Checking results daily and stopping when they look good is the other half of the problem. Early results swing widely. Stop at a high point and you'll declare winners that don't hold up in roll-out.

The winner's curse

Even properly run tests that pass significance tend to overstate the effect, especially when they're underpowered. The roll-out improvement is usually smaller than the test result.

Illustrative: the range of likely effects from a readout, and the chance the change helps.-15.0%-10.0%-5.0%0.0%5.0%10.0%15.0%Lift in weekly opens (%)
The range of likely effects from a readout, and the chance the change helps.

What good practice looks like

  1. Size every test before launch, from the baseline rate and the smallest effect that matters.
  2. Fix the duration, in whole weeks, and read out once.
  3. Report the effect with its range, not just significant or not. "Most likely +4%, somewhere between −1% and +9%" is more useful than a p-value.
  4. Keep a log of every test, including the inconclusive ones. They tell you which ideas aren't worth the traffic.

When traffic is limited

  • Test bigger changes: a new page or offer, not a button colour.
  • Test where traffic is highest: earlier steps in the funnel.
  • Use a nearer outcome: add-to-basket rather than purchase, validated later.
  • Combine research with fewer, better tests: user research tells you what's worth testing.

The aim isn't more tests. It's tests that can actually answer the question you're asking.

Experiments that tell you the truth about your site

We run research-led testing programmes on your site and funnels, designed to detect real effects and read out without wishful thinking.