Validation is one of the six Build-phase stages in tbl_Stages (between Optimization and Insights & Storyboarding in the protocol), and it earns that status because of scale. The modelling tool does not fit one model per cell - it fits every admissible combination of transformed variables across the adstock/Hill hyperparameter grid (Pre-Read slide 14, stages 5-6), storing validation and diagnostic statistics for each candidate. The structural evidence sits in the sample exports: a qc_file at model × variable grain (551 rows × 27 diagnostic columns in the reference archive) and a score_df composite scorecard (21 candidate models × 30 metric columns). Hundreds of candidates get scored mechanically; a ranked shortlist of survivors reaches a human; the human adjudicates against business knowledge. Your job in this module is to become that human.

The tool's own ranking order is visible in the production config's priority_order key - eleven metrics, earlier means higher priority: mape, mape_test, adj_r_squared, r_squared, dw_stat, shapiro_stat, shapiro_p_value, high_vif_count, low_t_stat_count, bp_stat, bp_pval. Read it as: accuracy first (train then test error), then fit, then residual diagnostics, then multicollinearity and significance counts as tie-breakers.

Two thresholds are stated explicitly on Pre-Read slide 14's selection criteria; the rest are named tests whose working cutoffs live in team practice.

Adjusted R-squared ≥ 80%

Measures how much of the sales variance the model explains, penalised for every extra variable added. Failure looks like a model that tracks the broad level of sales but misses its movements - the fitted line is a smoothed caricature of the actual series. The intuition: below the bar, too much of what the client cares about (the ups and downs they want explained) is sitting in the residual, so any decomposition you present is built on sand. The adjustment matters because MMM models carry many variables - raw R² can be bought by throwing regressors at the problem; adjusted R² cannot.

MAPE ≤ 10%

Mean Absolute Percentage Error - the average miss between predicted and actual sales, in percentage terms, evaluated on both train and held-out test data (the pipeline runs an 80-20 split precisely so MAPE on unseen weeks can be checked). Failure looks like a model that explains the past but cannot predict: fine in-sample, wild out-of-sample. The intuition: if the model misses actuals by more than 10% on average, the simulated scenarios built on it in module 3.3 inherit at least that error, and the client will notice.

Durbin-Watson (autocorrelation)

Tests whether residuals are correlated with their own past - whether today's miss predicts tomorrow's miss. The statistic sits near 2 when residuals are clean; values well below 2 signal positive autocorrelation (misses come in runs), well above 2 negative autocorrelation (misses alternate). Failure typically means a missing slow-moving driver: an unmodelled trend, a seasonality left in the error, a distribution change nobody gave the model. The intuition: a good model's errors should look like coin flips; if you can see a pattern in the residual plot, the model is leaving explainable structure on the table - and in this program's UCM world, that usually means the state components have not absorbed what they should.

Breusch-Pagan (heteroskedasticity)

Tests whether the residual variance is constant across the series, which is what slide 14 means by requiring "homoscedastic errors". A small BP p-value rejects constant variance: the model is precise in quiet periods and erratic in big ones (or vice versa). Failure looks like a residual plot that funnels - tight on the left, wide on the right, often because effects that scale with sales were modelled additively. The intuition: non-constant variance does not bias the coefficients themselves, but it corrupts their standard errors, which means every significance claim you make downstream is quietly wrong.

p-values and t-stats (significance)

Per-variable checks that each estimated coefficient is distinguishable from zero given the noise. A low t-stat / high p-value says the data cannot tell whether that channel does anything at all - its contribution could plausibly be zero. Failure looks like a model whose ROI story rests on statistically hollow coefficients. One program nuance carried from the KT sessions: the client-facing surface shows R²/adjusted R²/MAPE, not p-values - significance is an internal quality gate (the low_t_stat_count metric in priority_order), not a client talking point.

VIF (multicollinearity)

Variance Inflation Factor - measures how predictable each regressor is from the other regressors. High VIF means two or more variables move together so tightly (TV and Digital Video flighted in the same bursts, own price and competitor price tracking each other) that the model cannot apportion credit between them; coefficients become unstable and can flip sign between candidate models. Failure looks like implausible, unstable individual contributions inside a plausible total. The intuition: the model's total prediction can be excellent while its internal split is arbitrary - and MMM's entire product is the internal split. The scorecard tracks this as high_vif_count: how many variables in the candidate breach the threshold.

A model can pass every statistical gate and still be wrong - statistically clean nonsense is the most dangerous failure mode in MMM, because nothing mechanical catches it. Slide 14's selection criteria say it plainly: contribution and ROI must "align with business expectations". The validation bootcamp session (Pijush) breaks that into five working lenses:

• Topline validation with contribution bands. Each driver's contribution % must land inside the benchmark band set during bound-setting (Track 1.6/1.7) - distribution in its correlation-implied range, promo incrementality under its cap, media contributions consistent with ASSR. A model that says TV drives 40% of volume for a brand whose entire media budget is 3% of revenue fails here regardless of its R².
• ROI story alignment. Channel ROIs must respect the expected ROI hierarchy and each channel's benchmark band. An outlier ROI is not automatically wrong - but it must be explainable, and the explanation has to survive contact with the client's media team.
• Due-to story coherence. The year-over-year due-to bridge must tell a story the business recognises: if the client cut TV 30% and the due-to table shows TV adding growth, something upstream is broken. Contribution and due-to must also be internally consistent with each other (module 0.3's distinction).
• CPM alignment. Cost per mille per channel, computed from the model's spend and support inputs, should sit near known market rates. A CPM ten times the market rate usually means an input-data unit error (impressions in thousands vs units) that the fit statistics are blind to.
• mROI sanity. Marginal ROIs read off the response curves must decline with spend and sit in a believable relationship to average ROI (mROI below ROI for a saturating channel). An mROI above its own ROI at current spend, or one that increases with spend, points at a broken saturation fit.

Below is a synthetic score_df-style shortlist - ten candidate models for one imaginary cell, every number fabricated for teaching. The business bands for this exercise: TV contribution expected 8-14%, TV ROI expected 0.8-1.6. Apply the gates: adj R² ≥ 80%, MAPE ≤ 10%, DW roughly 1.5-2.5, BP p > 0.05, VIF < 10, business KPIs in band. Work the table before reading the row colours.

C-01 passes cleanly. C-09 passes every gate but hugs four of them at once (MAPE, VIF, DW, and both business bands at their edges) - a reviewer keeps it on the shortlist but would not present it without a second look at its residual plot. Everything red fails at least one named gate; C-07 and C-08 are the instructive ones, because they fail on business criteria alone with immaculate statistics. C-10 fails four ways simultaneously - in a real score_df it would never reach the human, which is the point of mechanical ranking.