# MixLift Model Quality Documentation

**Last updated:** 2026-03-07
**Owner:** Data Science Manager

This document describes MixLift's modeling approach, default configuration, validation
methods, and known limitations. It supports Sales demos (credibility), Marketing content
(technical claims), and Customer Success (interpreting results).

---

## 1. Model Architecture

MixLift uses **PyMC-Marketing's Bayesian Media Mix Model (MMM)** with the following components:

- **Adstock:** Geometric decay (single parameter `alpha` per channel). Models the lagged
  effect of advertising — a dollar spent this week continues influencing revenue in subsequent
  weeks, with geometrically decaying impact.
- **Saturation:** Logistic saturation (parameter `lam` per channel, scaling `beta`). Models
  diminishing returns — doubling spend does not double revenue. The logistic function
  `beta * (1 - exp(-lam * x)) / (1 + exp(-lam * x))` maps spend to response.
- **Seasonality:** 6 Fourier terms for yearly seasonality, plus explicit Q4 indicator and
  cyclical month encoding (sin/cos). Captures DTC retail's holiday patterns.
- **Controls:** Trend (linear), holiday indicator (US holidays auto-detected via `holidays`
  package), month cyclical encoding.
- **Intercept:** Captures baseline/organic revenue from non-paid sources (brand, SEO, direct
  traffic, word-of-mouth).

All inference is fully Bayesian via MCMC (No-U-Turn Sampler / NUTS).

---

## 2. Default Priors

PyMC-Marketing sets the following default priors (we use their defaults without override):

| Parameter | Prior | Rationale |
|-----------|-------|-----------|
| `saturation_lam` (per channel) | `Gamma(alpha=3, beta=1)` | Weakly informative; allows the data to determine saturation speed. Centers around lam=3, which means moderate saturation. |
| `saturation_beta` (per channel) | `HalfNormal(sigma=2)` | Positive scaling factor. HalfNormal ensures positivity while allowing wide range. |
| `adstock_alpha` (per channel) | `Beta(alpha=1, beta=3)` | Favors lower retention rates (mode ~0.0, mean ~0.25). Most channels have relatively fast decay. |
| `intercept` | `Normal(mu=0, sigma=large)` | Weakly informative. Estimated from data scaling. |
| `sigma` (noise) | `HalfNormal` | Observation noise. Weakly informative. |
| Fourier coefficients | `Normal(0, sigma)` | Standard weakly informative priors for seasonality. |

**Why these priors:** PyMC-Marketing's defaults are designed for typical marketing data.
They are weakly informative — strong enough to regularize with short data (preventing
nonsensical estimates) but weak enough to let the data speak with 26+ weeks of observations.
We deliberately do not override them because:

1. Custom priors require domain expertise about each customer's specific channels
2. Default priors produce reasonable results across the DTC eCommerce ICP ($5M-$50M revenue)
3. Overriding priors introduces a maintenance burden and a "tuning knob" that could silently degrade results

**Prior sensitivity with short data:** With fewer than 26 weeks of data, the posterior
is more influenced by these priors than by the data itself. MixLift flags this with an
"Exploratory Analysis" badge. Results should be treated as directional, not precise.

---

## 3. MCMC Configuration

| Parameter | Default | MCP Server | Rationale |
|-----------|---------|------------|-----------|
| Chains | 4 | 4 | Required for reliable R-hat diagnostics. Fewer chains mask convergence issues. |
| Tune | 1,000 | 500 | Warmup iterations. MCP uses 500 for speed; default uses 1,000 for quality. |
| Draws | 500 | 250 | Posterior samples per chain. MCP uses 250 (1,000 total across 4 chains). |
| target_accept | 0.90 | 0.90 | Acceptance rate for NUTS. 0.90 reduces divergent transitions while maintaining sampling efficiency. |
| Cores | 1 | 1 | Sequential chains to avoid macOS ARM multiprocessing deadlocks. |

**Why no fixed random seed:** Fixed seeds mask sampling variability and prevent convergence
diagnostics (R-hat) from being truly diagnostic. Each run explores independent regions of
the posterior, giving honest uncertainty estimates.

---

## 4. Convergence Diagnostics

### R-hat (Gelman-Rubin Statistic)

R-hat measures between-chain vs. within-chain variance. Values near 1.0 indicate convergence.

| R-hat Value | Interpretation |
|-------------|---------------|
| < 1.01 | Excellent convergence |
| 1.01 - 1.05 | Acceptable convergence |
| > 1.05 | **Warning:** chains have not converged. Estimates may be unreliable. |

MixLift checks R-hat for all model parameters after fitting. Any parameter with R-hat > 1.05
triggers a convergence warning in the output. The recommended action is to increase `tune`
and `draws` or check the data for anomalies.

### Divergent Transitions

Divergent transitions indicate the sampler encountered regions of high curvature in the
posterior that it could not navigate accurately. They signal potential bias in the posterior
estimates.

| Divergences | Interpretation |
|-------------|---------------|
| 0 | No issues |
| 1-10 (< 1% of samples) | Minor; results are likely fine but worth noting |
| > 1% of samples | **Warning:** model may need reparameterization or higher `target_accept` |

---

## 5. Posterior Predictive Checks

MixLift runs `sample_posterior_predictive()` after model fitting. This generates simulated
data from the fitted model and compares it to the actual observed data.

**Metrics computed:**

- **R-squared:** Proportion of variance explained. Reported as "This model explains X% of
  your revenue variation." Values above 80% are typical for well-specified MMMs with
  sufficient data.
- **MAPE (Mean Absolute Percentage Error):** Average prediction error as a percentage of
  actual values. Lower is better; < 15% is good for MMM.

If posterior predictive sampling fails (rare), MixLift warns the user and continues with
the analysis. The failure is non-blocking because ROAS and contribution estimates come from
the posterior, not the posterior predictive.

---

## 6. LOO-CV (Leave-One-Out Cross-Validation)

LOO-CV estimates out-of-sample predictive accuracy using Pareto-Smoothed Importance Sampling
(PSIS-LOO) via ArviZ. It answers: "How well would this model predict data it hasn't seen?"

### Key Metrics

| Metric | Description |
|--------|-------------|
| **ELPD-LOO** | Expected Log Pointwise Predictive Density. Higher (less negative) is better. Comparable across models on the same dataset. |
| **SE** | Standard error of the ELPD estimate. |
| **p_loo** | Effective number of parameters. Should be less than the actual number of data points. High p_loo suggests overfitting. |
| **Pareto k** | Per-observation diagnostic. k > 0.7 means that observation is highly influential and the LOO estimate may be unreliable for it. |

### Interpretation Guide

| Interpretation | Criteria | What It Means |
|---------------|----------|---------------|
| **Good fit** | No Pareto k > 0.7, p_loo < 40% of n_data_points | Model generalizes well. Predictions are reliable. |
| **Fair fit** | Some Pareto k > 0.7 (< 10% of observations), p_loo reasonable | Model is usable but some data points are poorly predicted. May indicate outliers or structural breaks in the data. |
| **Poor fit** | Many Pareto k > 0.7 (>= 10%) or p_loo >= 40% of data points | Model may be overfitting or misspecified. Consider: more data, fewer channels, or checking for data quality issues. |

### Typical ELPD Ranges for MMM

ELPD is on a log-likelihood scale and depends on the scale of the target variable. It is
not directly interpretable in isolation — its value is in **comparing models** on the same
dataset. A model with higher ELPD-LOO is preferred.

For typical DTC eCommerce weekly revenue data (52 weeks, 4-8 channels):
- ELPD-LOO / n in the range of -2 to -5 per observation is typical
- p_loo of 10-30 is normal (reflects the number of effective model parameters)

---

## 7. Intercept Decomposition (Baseline/Organic Revenue)

The model intercept represents revenue that would occur with zero ad spend. MixLift
extracts and presents this as "estimated organic/baseline revenue."

**Math:**
- `weekly_organic_revenue = E[intercept]` (posterior mean)
- `annualized_organic_revenue = weekly_organic_revenue * 52`
- Confidence interval: 5th and 95th percentiles of the posterior intercept

**What it captures:** Brand equity, SEO traffic, direct/type-in traffic, word-of-mouth,
email list revenue, and any other revenue source not driven by the paid channels in the
model.

**Caveat:** If an important paid channel is missing from the data, its effect will be
absorbed into the intercept, inflating the organic revenue estimate.

---

## 8. Known Limitations

### 8.1 Short Data (< 26 weeks)

With fewer than 26 weeks of data:
- Posterior is dominated by priors rather than data
- Seasonality cannot be reliably estimated (need 52+ weeks for annual patterns)
- Adstock effects are harder to identify (need enough time variation in spend)
- MixLift flags this with an "Exploratory Analysis" badge

**Recommendation:** 52 weeks minimum for production-grade estimates. 26-51 weeks is usable
but estimates carry wider uncertainty. Below 26 weeks, treat results as directional only.

### 8.2 Cross-Channel Interactions

MixLift models each channel independently — it does not capture interactions like:
- Meta prospecting driving Google branded search
- YouTube awareness boosting Meta retargeting
- Email conversions attributed to email but driven by paid social

This is a fundamental limitation of additive MMM. The practical impact is that channels
which drive top-of-funnel awareness may appear to have lower ROAS than their true causal
contribution, while bottom-of-funnel channels (branded search, retargeting, email) may
appear inflated.

**Mitigation:** Interpret ROAS comparisons between top-of-funnel and bottom-of-funnel
channels with caution. The budget optimizer accounts for marginal ROAS at current spend
levels, which partially mitigates this issue.

### 8.3 Point-Estimate Budget Optimizer

The budget optimizer uses posterior mean parameters (point estimates) to find the optimal
allocation. It does not propagate full posterior uncertainty through the optimization.

**Impact:** The "optimal" allocation is the best single guess, but the true optimal could
differ. Channels with wide posterior uncertainty may have optimistic or pessimistic
allocations.

**Mitigation:** Scenario mode uses full posterior sampling for impact projections, so
users can test specific reallocation ideas with proper uncertainty bounds. The confidence
summary flags channels with low confidence.

### 8.4 Stationarity Assumption

The model assumes that the relationship between spend and revenue is constant over the
data period. If the market changed significantly (new competitor, brand crisis, COVID),
the model blends pre- and post-change periods.

**Recommendation:** Use data from a single "regime" if possible. If a major change occurred
mid-period, consider using only post-change data (accepting shorter data as a tradeoff).

### 8.5 Attribution vs. Incrementality

MMM estimates the *correlation* between spend changes and revenue changes, controlling for
other channels and seasonality. It does not establish causality with the same rigor as a
randomized experiment (geo-lift test, holdout test).

**Recommendation:** For high-stakes decisions (killing a channel entirely), validate MMM
findings with a holdout experiment. For budget rebalancing (shifting 10-20% between channels),
MMM guidance is typically sufficient.

---

## 9. Quality Checklist for Interpreting Results

Before acting on MixLift results, verify:

1. **Convergence:** No R-hat warnings, no divergent transitions
2. **Model fit:** R-squared > 70% (ideally > 85%)
3. **LOO-CV:** "Good" or "Fair" interpretation
4. **Data length:** 26+ weeks (52+ preferred)
5. **ROAS plausibility:** No channel showing ROAS > 20x or < 0 (sanity check)
6. **Confidence:** Check per-channel confidence summary; act cautiously on "low confidence" channels
7. **Organic baseline:** Verify the intercept-derived organic revenue is plausible for the brand

---

*This document is maintained by the Data Science Manager. Last reviewed 2026-03-07.*
