Common Failure Modes and Troubleshooting
Track 3 - Advanced · Module 3.4
All analysts
Course home
Learning ObjectivesModule 3.4 · ~30 min
Recognise the symptom pattern of the ten recurring failure modes the program's trainers flag across the KT sessions.
Diagnose which failure mode a broken read points to, and state the trainer's fix rule from memory.
Know which KT session or reference sheet to consult when one of these surfaces in a live cell.
Where this list comes from

Every KT session ended up part methodology, part cautionary tale - the trainers kept interrupting their own walkthroughs with "and here is where people get burned". This module compiles the ten most instructive of those warnings, drawn from all five sessions (2 Jun - 25 Jun) plus the config-mapping pitfalls documented in the Methodology workbook's Config Creation sheet and observed in a real production YAML. Each card has three parts: what the failure looks like from the outside, what actually went wrong underneath, and the rule that fixes it. The closing quiz gives you six synthetic incidents to diagnose cold - the skill this module is actually for.

Data assembly traps

1. Adding e-commerce sales to offline sales

What it looks like: a "total brand" sales series built as offline plus e-commerce that runs a few percent hotter than any syndicated brand read; contributions that refuse to reconcile between the offline and e-com models.

What actually went wrong: in markets where retailers run their own online and click-and-collect operations (the UK and US pattern), the offline Nielsen read already contains e-commerce sales. Adding the e-com feed on top double counts them. "You cannot do e-com plus offline to get to the entire brand sales."

The fix: a separate offline read and a separate e-com read, never additive. E-com media is tagged against e-com sales so the comparison stays apples to apples, and the client validates the retailer list per market so you know exactly where the overlap sits. (2 Jun data-onboarding session.)

2. The Nielsen/Kantar merge trap

What it looks like: Kantar volumes merged wholesale into Nielsen "for completeness", or hard and soft discounters modelled as their own variables; a sales spine that no longer ties to either source.

What actually went wrong: Kantar's genuine contribution is the hard- and soft-discounter retailers excluded from the Nielsen universe - typically ~2% of volume - and some other retailers are tracked by both sources, so an indiscriminate merge double counts the overlap while the discounter slice adds almost nothing.

The fix: sum hard plus soft discounters (never model them separately) and add them to Nielsen only where the share is material. If Nielsen already covers ~99.9%, skip the merge entirely and tell the client - "it doesn't add too much value rather than it makes it more difficult". (2 Jun session.)

3. Promo volume as a driver of volume

What it looks like: a promo variable with a near-perfect fit that the model leans on for everything; suspiciously beautiful diagnostics on the first run.

What actually went wrong: promo sales volume is a component of the volume KPI - regressing the target on a slice of itself is circular, and the near-perfect correlation is the giveaway. The program dropped promo volume as a variable for exactly this reason.

The fix: the replacement hierarchy - request trade spends first; if unavailable, use the promo TDP index and promo price index (indexed measures of promo depth, not absolutes). And carry the structural guard: promo TDP contribution can never exceed the total distribution contribution, because promo TDP is a subset of TDP. (2 Jun session; bounds treatment in the 15 Jun session.)

Variable construction traps

4. Reading the raw competitor-price correlation literally

What it looks like: a competitor-price bound sized straight off an observed correlation with own volume - sometimes larger than the own-price impact itself.

What actually went wrong: the competitor series is the top four or five competitors aggregated into one average, and the correlation of that blend against your volume is spurious - it reflects the mix, not a causal price response. The same lesson applies to own price expressed as an index: an index masks how fast you dropped price versus how fast the competitor did, a failure the team was burned by in a live market before falling back to absolutes.

The fix: enter competitor average price as its own absolute variable with a positive sign, and size it as a relative measure - a share of your own price contribution: at most ~50% if you lead the category, ~70% if you are a weak player, potentially above 50% mid-pack. Sanity-check by eyeballing own price, competitor price, and own volume year by year, and remember the guard: "if your price doesn't matter, your competitor price also doesn't really matter." (15 Jun bound-setting session.)

5. Halo masquerading as direct media - and vice versa

What it looks like: master-brand media split across variants by sales share, with every variant model then reporting brand media as its top driver; or a new variant's media tested on total-brand sales and labelled a "halo".

What actually went wrong: mappability was ignored. A halo runs only from media tagged to one product onto a different product's sales. Master-brand media is not a halo on anything - it is funnel activity for every variant that carries the brand, and distributing it across variants invents precision the data does not have. Conversely, a variant whose sales already sit inside the total cannot halo that total; its media is a direct feature on the total until the core is carved out.

The fix: let media mappability set the model grain. Master-brand media enters as its own feature, with the standing check that brand media impact stays below direct media impact. True halos (variant media onto core) are sized at 5-10% of that media's contribution in its own donor model, and a halo can never exceed the direct media. (2 Jun session; halo rule book in the 25 Jun session.)

6. Trusting brand-level MDS for variant reads

What it looks like: a recently launched variant showing a strong salience or meaningfulness contribution in its long-term model - implausibly mature brand equity for its age.

What actually went wrong: the Kantar brand-health (BTS) tracker exists at brand level only and is duplicated down to every sub-brand and variant. The young variant inherited the parent brand's indexes wholesale - "it actually misleads sometimes", in the trainer's words, because that is the only brand-health repository there is.

The fix: there is no data fix - it is a disclosure rule. Flag the limitation to the client explicitly, put it in the caveats, and treat variant-level long-term reads as directional. Module 2.4 covers the construct this feeds. (2 Jun session; reiterated in the 25 Jun session.)

Config traps

7. Keyword collisions in platform-category mapping

What it looks like: one platform's spend total silently doubling while a sibling platform reads near zero; columns landing in the wrong L1 category with no error raised anywhere.

What actually went wrong: the config's platform_categories block matches keywords against ADS column names, and matching is exact on every character - whitespace, underscores, case. Substring collisions are the classic killer: a retailer token that is also the prefix of two other retailers' longer names sweeps their columns into its category. Generic and brand-specific categories that coexist (a generic social category alongside a brand-specific one) shadow each other the same way. And the semantics differ by block - the platform categories and halo_column_keyword are keyword-matched, while neg_col and the scaling lists are exact-name - so a fragment placed in the wrong kind of list silently no-ops.

The fix: rename the colliding ADS columns to unique, collision-free tokens before config work; keep every keyword unique per category; use platform_categories_no_keywords for exclusions; and run the Config Creation checklist - no empty lists in the L1/L2/L3 mapping, case checked, file saved. (Methodology workbook, Config Creation sheet; production YAML structural notes.)

8. Copying a config that carries stale pins

What it looks like: a brand-new market whose TV adstock grid contains several many-decimal values, or long-term bounds where LB equals UB exactly - the model "converging" instantly to oddly specific answers.

What actually went wrong: configs are built by copying the most recent one, and the copy carried the previous market's fitted optima pinned back into its parameter grids. The alpha_range/beta_range keys are not ranges at all but discrete candidate grids - the search only ever tries the listed values, so a stale pinned value quietly becomes the answer.

The fix: treat every copied config as contaminated until proven otherwise. Rebuild the parameter grids from the platform-category benchmark bands for the new market, and pin a bound (LB = UB) only as a deliberate, documented act. (Production YAML structural notes; Config Creation sheet.)

Interpretation traps

9. The radio burst-spend trap

What it looks like: a small channel that naive arithmetic says must be a star - spend a tenth of TV's, CPM a quarter of TV's, so surely a far higher ROI - and an analyst about to hand it a generous bound.

What actually went wrong: the yearly totals hid the execution pattern. All of the channel's spend sat in one burst early in the window, with nothing since - so there is no carry-through in the analysis period at all. Mechanically, when spend collapses much faster than CPM, ROI inflates by arithmetic alone: impressions are spend over CPM, so the denominator vanished while the numerator only sagged.

The fix: always inspect the weekly execution time series, never just yearly totals. Compare the drop in spend against the drop in CPM - if they are proportional the effects negate; if spend fell much harder, the "great ROI" is an artefact. Systematic investment earns a real bound; burst spending does not. (23 Jun media bound-setting session.)

10. Extrapolating a tiny channel's high ROI

What it looks like: a recommendation slide that says "shift budget into print/OOH because its ROI is the highest in the mix".

What actually went wrong: a channel with tiny spend sits at the very start of its saturation curve, where marginal returns are at their peak - and often the spend was sporadic or a trial to begin with. Its measured ROI says nothing about what the tenth incremental unit of spend would return at scale.

The fix: channel-mix recommendations come from simulation and optimisation over the response curves, never from ranking measured ROIs. Check the channel's position on its curve, the consistency of its spend, and the secondary research on what media that market's consumers actually consume before any reallocation claim. (23 Jun session; response-curve mechanics in module 3.3.)

The meta-pattern: nearly every trap on this page is a plausible number trusted without its context - a correlation without its aggregation, an ROI without its spend pattern, a keyword without its collisions, a feed without its overlap. The trainers' habit worth copying is not scepticism of numbers but the reflex of asking what produced them.
Check Yourself: what went wrong?
Vignette 1: an analyst builds "total brand sales" for a UK cell by summing the offline sales feed and the e-commerce sales feed. The total comes out ~5% above the syndicated brand read. Which failure mode is this?
Why: in click-and-collect markets the offline read already contains e-com sales; adding the e-com feed double counts them. The rule: separate offline read, separate e-com read, never additive.
Vignette 2: a brand runs media at total-brand level only. The analyst distributes that spend across its three formats by volume share, and each format model then shows "its" media as the biggest driver. What went wrong?
Why: media that cannot be mapped below brand level is master-brand media. Splitting it by sales share invents variant-level precision the data does not have; the fix is one shared feature per model with the brand-media-below-direct-media check.
Vignette 3: a channel spent only in the first year of a three-year window and has been dark since. Its CPM is a quarter of TV's and its spend a tenth, so the analyst pencils in an ROI bound well above TV's. What went wrong?
Why: yearly totals hid a burst execution pattern. When the spend denominator collapses faster than the CPM-driven numerator, ROI inflates by arithmetic alone - inspect the weekly execution chart before granting any bound.
Vignette 4: after new retailer columns land in the ADS, one e-commerce platform's mapped spend doubles overnight while two sibling retailers read zero. The model ran without a single error. What went wrong?
Why: platform-category mapping is substring-based and raises no errors. A short retailer token that prefixes two longer retailer names captures their columns; the fix is renaming ADS columns to collision-free tokens before config work.
Vignette 5: a variant launched 14 months ago shows a salience index far above the category average, and its long-term model hands salience a hefty contribution. What went wrong?
Why: the Kantar tracker is captured at brand level and duplicated to variants; a 14-month-old variant cannot genuinely carry that salience. No data fix exists - caveat it to the client and treat the variant LT read as directional.
Vignette 6: the aggregated competitor-average-price series shows a tidy negative correlation with own volume, and the analyst sizes competitor price to a larger contribution than own price. What went wrong?
Why: the competitor series blends several competitors, so its raw correlation is spurious. The rule is relative: competitor price takes at most ~50% of your own price impact if you lead, ~70% if you are weak - and if your own price does not matter, the competitor's cannot either.
Sources
Authored from:
  • UL - KT.docx (2 Jun data-onboarding session) - e-com/offline overlap, Nielsen/Kantar merge rule, promo-volume drop and trade-spend hierarchy, master-brand media rule, BTS granularity caveat
  • UL - KT (2).docx (15 Jun session) - competitor-price relative-measure rule, spurious aggregated correlations, price index vs absolutes, promo TDP subset guard
  • UL - KT (3).docx (23 Jun session) - the radio burst-spend trap, spend-vs-CPM drop comparison, saturation-curve position rule, secondary-research leg
  • UL - KT (4).docx (25 Jun session) - halo rule book (5-10% of donor model, halo below direct media), master-brand-is-not-halo distinction
  • MathCo Methodology Understanding_UL.xlsx sheet Config Creation - keyword-matching pitfalls, column-renaming fix, quick-checks list
  • Production MMX config YAML (structural notes) - discrete parameter grids vs "ranges", stale pinned optima, keyword vs exact-name semantics, category shadowing
All vignettes and every number in them are synthetic; the rules quoted are the trainers' stated methodology, not client data.