Skip to content
home

Measurement strategy · 2 min read

Geo holdout tests: a practical guide

From candidate regions to readout, step by step.

By WebIQ Analytics · 11 December 2025

A geo holdout test changes spend in some regions and not others, then compares the two. It's the most direct way most businesses can measure what a channel actually adds, without relying on user-level tracking.

1. Start from the decision

Write down what you'd do differently depending on the result, and the smallest effect that would change it. That sets how sensitive the test must be.

2. Choose regions you can match

Build a comparison for each candidate test region from other regions that tracked it closely before the test (a synthetic control). Rank candidates by how well they can be matched and choose from the top.

3. Check the test can detect the effect

Estimate the smallest effect the design can detect at different lengths, using your own data. If it's larger than the effect that matters, run longer, add regions or test a bigger change.

4. Run placebo checks

Apply the same analysis to regions you won't touch. If they show effects as large as the one you're looking for, the design will mistake noise for results.

Illustrative: observed results against a synthetic comparison after the test starts.100110120130W1W5W9W13W17W21W25Weekly orders (k)Test start
Observed results against a synthetic comparison after the test starts.

5. Agree the rules and launch

Fix the regions, start date, minimum duration and analysis before launch. Confirm platforms can change spend by region, and hold national activity equal.

6. Monitor health, not results

Each week check delivery, data arrival and contamination. Don't read the result until the agreed date.

7. Read out once

Report the effect with its range and the placebo comparison, in plain language: what changed, how sure you are, and what to do next.

8. Feed it back

Use the result to calibrate your mix model for that channel. The model improves, and the next test goes where it's still least certain.

Four practice areas, one aim

Each service stands alone, and they work best together: clean data, sound models, honest tests.