A common pattern: a team runs a test for two weeks, sees no significant result, concludes "it made no difference", and moves on. Often the test never had a realistic chance of detecting the effect.
Underpowered by design
If a page converts at 2% and the test aims to spot a 5% relative improvement, it needs hundreds of thousands of visitors per variant. Run it on a fraction of that and "no significant difference" means "we couldn't tell", not "there's no effect".
Peeking
Checking results daily and stopping when they look good is the other half of the problem. Early results swing widely. Stop at a high point and you'll declare winners that don't hold up in roll-out.
The winner's curse
Even properly run tests that pass significance tend to overstate the effect, especially when they're underpowered. The roll-out improvement is usually smaller than the test result.
What good practice looks like
- Size every test before launch, from the baseline rate and the smallest effect that matters.
- Fix the duration, in whole weeks, and read out once.
- Report the effect with its range, not just significant or not. "Most likely +4%, somewhere between −1% and +9%" is more useful than a p-value.
- Keep a log of every test, including the inconclusive ones. They tell you which ideas aren't worth the traffic.
When traffic is limited
- Test bigger changes: a new page or offer, not a button colour.
- Test where traffic is highest: earlier steps in the funnel.
- Use a nearer outcome: add-to-basket rather than purchase, validated later.
- Combine research with fewer, better tests: user research tells you what's worth testing.
The aim isn't more tests. It's tests that can actually answer the question you're asking.
