A geo holdout test changes spend in some regions and not others, then compares the two. It's the most direct way most businesses can measure what a channel actually adds, without relying on user-level tracking.
1. Start from the decision
Write down what you'd do differently depending on the result, and the smallest effect that would change it. That sets how sensitive the test must be.
2. Choose regions you can match
Build a comparison for each candidate test region from other regions that tracked it closely before the test (a synthetic control). Rank candidates by how well they can be matched and choose from the top.
3. Check the test can detect the effect
Estimate the smallest effect the design can detect at different lengths, using your own data. If it's larger than the effect that matters, run longer, add regions or test a bigger change.
4. Run placebo checks
Apply the same analysis to regions you won't touch. If they show effects as large as the one you're looking for, the design will mistake noise for results.
5. Agree the rules and launch
Fix the regions, start date, minimum duration and analysis before launch. Confirm platforms can change spend by region, and hold national activity equal.
6. Monitor health, not results
Each week check delivery, data arrival and contamination. Don't read the result until the agreed date.
7. Read out once
Report the effect with its range and the placebo comparison, in plain language: what changed, how sure you are, and what to do next.
8. Feed it back
Use the result to calibrate your mix model for that channel. The model improves, and the next test goes where it's still least certain.
