A mix model that fits the past closely isn't necessarily right. With enough variables, any model can match history. Validation is how you find out whether it has learned something real.
1. Accuracy on weeks it didn't see
Hold back the most recent weeks (often 8 to 13), fit the model on the rest, and compare its predictions with what actually happened. Report the error, and check the misses aren't all in one direction.
2. Technical health
Bayesian models are estimated by sampling. Check that sampling converged: standard diagnostics should be within accepted limits, with no warnings left unexplained. A model that didn't converge hasn't produced reliable estimates, however good its charts look.
3. Plausible values
Look at each channel's return, carryover and saturation. Are they believable for that kind of channel? A paid search effect lasting six months, or a channel with a return of £20 per £1, needs a very good explanation.
4. Stability
Refit with a few weeks more or less data, or slightly different settings. Results that swing dramatically aren't reliable enough to move budget on.
5. Agreement with experiments
Compare the model's estimate for a channel with any geo test or lift study for that channel. Broad agreement builds confidence. Disagreement means one of them is wrong; usually the test is better evidence, and the model should be recalibrated.
6. Recovering a known truth
The strongest check of a modelling approach is to run it on simulated data where the true effects are known, and see whether it recovers them. It shows what the method can and can't separate with data like yours.
What to do with the results
Write a short validation note alongside the readout: what was checked, what passed, and what remains uncertain. It's what turns a model into evidence finance can accept.
Red flags
- Only in-sample fit is reported.
- Results change completely between refreshes.
- Every channel looks profitable.
- Nobody can say what the priors were.
