A test ends and the tool shows a number: "+4% conversion, 95% significant". Deciding what to do with it needs a little more than that.
Look at the range, not just the headline
The estimated lift is the middle of a range of plausible values. Read the whole range:
- Range entirely above zero: the change very likely helps.
- Range entirely below zero: it very likely hurts.
- Range spanning zero: you can't tell. That's inconclusive, not "no effect".
- Range above zero but tiny: real, but possibly too small to be worth the cost of rolling out and maintaining.
Check it reached the planned sample
A result read before the planned sample was reached is unreliable, however significant it looks. Early results swing, and stopping at a good moment inflates false wins.
Check guardrail metrics
A change that lifts clicks but lowers average order value, or raises returns, may not be a win. Agree two or three guardrail metrics before launch and check them at readout.
Be wary of segment fishing
After an inconclusive overall result, it's tempting to look for a segment where the change "worked": mobile users, returning visitors, one region. With enough segments, something will look significant by chance. Treat segment findings as hypotheses for a new test, not conclusions.
Expect the effect to shrink
Winning tests tend to overstate the true effect, especially smaller ones. Plan roll-outs expecting a somewhat smaller gain than the test showed, and monitor after launch.
Make a clear decision
Each test should end with one of:
- Ship: clear win on the primary metric, guardrails fine.
- Don't ship: clear loss, or guardrails harmed.
- Inconclusive: keep the control, record the learning, and decide whether a bolder version is worth testing.
Write the decision and the reason in your test log, alongside the hypothesis. That's how a programme learns rather than just runs tests.
