Skip to content
home

Marketing mix modelling · 2 min read

How to validate a mix model before you trust it

Six checks that separate a model you can act on from one that only fits the past.

By WebIQ Analytics · 16 July 2026

A mix model that fits the past closely isn't necessarily right. With enough variables, any model can match history. Validation is how you find out whether it has learned something real.

1. Accuracy on weeks it didn't see

Hold back the most recent weeks (often 8 to 13), fit the model on the rest, and compare its predictions with what actually happened. Report the error, and check the misses aren't all in one direction.

Illustrative: actual against fitted, including weeks the model didn't see.7008009001k1.1k1.2k7 Oct20 Jan5 May18 Aug1 Dec16 Mar29 JunWeekly revenue (£k)Holdout
Actual against fitted, including weeks the model didn't see.

2. Technical health

Bayesian models are estimated by sampling. Check that sampling converged: standard diagnostics should be within accepted limits, with no warnings left unexplained. A model that didn't converge hasn't produced reliable estimates, however good its charts look.

3. Plausible values

Look at each channel's return, carryover and saturation. Are they believable for that kind of channel? A paid search effect lasting six months, or a channel with a return of £20 per £1, needs a very good explanation.

4. Stability

Refit with a few weeks more or less data, or slightly different settings. Results that swing dramatically aren't reliable enough to move budget on.

5. Agreement with experiments

Compare the model's estimate for a channel with any geo test or lift study for that channel. Broad agreement builds confidence. Disagreement means one of them is wrong; usually the test is better evidence, and the model should be recalibrated.

6. Recovering a known truth

The strongest check of a modelling approach is to run it on simulated data where the true effects are known, and see whether it recovers them. It shows what the method can and can't separate with data like yours.

What to do with the results

Write a short validation note alongside the readout: what was checked, what passed, and what remains uncertain. It's what turns a model into evidence finance can accept.

Red flags

  • Only in-sample fit is reported.
  • Results change completely between refreshes.
  • Every channel looks profitable.
  • Nobody can say what the priors were.

Know what your marketing really delivers, and where the next pound should go

We build Bayesian mix models on your data, show where the evidence is strong and where it isn't, and turn the result into a budget decision finance will accept.