Skip to content
home

Conversion optimisation · 2 min read

Reading test results: lift, ranges and when to call it

How to interpret an A/B test result without over- or under-claiming.

By WebIQ Analytics · 28 May 2026

A test ends and the tool shows a number: "+4% conversion, 95% significant". Deciding what to do with it needs a little more than that.

Look at the range, not just the headline

The estimated lift is the middle of a range of plausible values. Read the whole range:

  • Range entirely above zero: the change very likely helps.
  • Range entirely below zero: it very likely hurts.
  • Range spanning zero: you can't tell. That's inconclusive, not "no effect".
  • Range above zero but tiny: real, but possibly too small to be worth the cost of rolling out and maintaining.
Four results read by their intervals.No differenceClear win+4.1%Clear loss−3.2%Inconclusive+1.0%Win, but too small to matter+0.6%Illustrative: the dot is the estimated lift; the bar is its likely range.
Four results read by their intervals.

Check it reached the planned sample

A result read before the planned sample was reached is unreliable, however significant it looks. Early results swing, and stopping at a good moment inflates false wins.

Check guardrail metrics

A change that lifts clicks but lowers average order value, or raises returns, may not be a win. Agree two or three guardrail metrics before launch and check them at readout.

Be wary of segment fishing

After an inconclusive overall result, it's tempting to look for a segment where the change "worked": mobile users, returning visitors, one region. With enough segments, something will look significant by chance. Treat segment findings as hypotheses for a new test, not conclusions.

Expect the effect to shrink

Winning tests tend to overstate the true effect, especially smaller ones. Plan roll-outs expecting a somewhat smaller gain than the test showed, and monitor after launch.

Make a clear decision

Each test should end with one of:

  1. Ship: clear win on the primary metric, guardrails fine.
  2. Don't ship: clear loss, or guardrails harmed.
  3. Inconclusive: keep the control, record the learning, and decide whether a bolder version is worth testing.

Write the decision and the reason in your test log, alongside the hypothesis. That's how a programme learns rather than just runs tests.

Experiments that tell you the truth about your site

We run research-led testing programmes on your site and funnels, designed to detect real effects and read out without wishful thinking.