Experiment 003
003 — Regression to the mean: how much do "worst" cohorts improve on their own?
Content optimization naturally targets the buyer questions where a brand performs worst. How much do those cohorts improve with no intervention at all?
Headline result: +19pp apparent win-rate gain from pure mean reversion, with zero real change.
Question. Content optimization naturally targets the buyer questions where a brand performs worst. Badly-performing samples of a noisy metric improve on re-measurement with no intervention at all — regression to the mean. How big is that effect here?
Why it matters. A random-cohort placebo (experiment 002) is centered on zero, so it can’t see this bias. But no real optimization program picks cohorts at random — it picks losers. If losers rebound on their own, every before/after comparison in this category is upward-biased by construction, and “we improved your worst questions” is the single easiest false claim to make.
Method
Same null window as experiment 002 (two runs, ten days apart, nothing shipped). Instead of random cohorts, draw 15-question cohorts from the worst-performing pool at baseline (bottom tier by win rate, or by visibility), run the same production estimator, and compare against random-cohort and best-performing-cohort draws. 200 cohorts per condition.
Results
| Cohort selection | Mean “effect” on the null window |
|---|---|
| Worst by win rate | +19.1pp |
| Worst by visibility | +17.7pp |
| Random | −1.1pp |
| Best by win rate | −13.7pp |
Textbook mean reversion, in both directions: cohorts selected for doing badly “gain” ~18–19 points with zero real change; cohorts selected for doing well “decline” ~14.
The nuance: what our product actually selects on
Our production pipeline selects treated questions by issue fingerprint (which feature, competitor, or topic a finding is about), not directly by performance rank. We replayed the real historical treated cohorts through the same null window: they averaged +2.75pp — near the random floor, far below the +19pp worst-selection bound. Performance selection enters indirectly (issues are minted where the brand loses), so the realistic bias sits between the floor and the bound, and we can measure it per cohort.
What we changed because of this
- Any claimed effect on performance-selected cohorts must beat the measured mean-reversion baseline for that selection rule — not zero.
- Sales and reporting language never uses “your worst questions improved” as evidence on its own; that outcome is the default expectation under no intervention.
- The selection-aware placebo joins the random placebo as a standing harness, so the bias is re-measured rather than remembered.
Takeaway for buyers of any tool in this category
Ask your vendor two questions: what does your estimator report on a window where nothing shipped? and how much do your targeted cohorts improve without intervention? If they can’t answer with numbers, their case studies are — statistically speaking — indistinguishable from weather.