# 003 — Regression to the mean: how much do "worst" cohorts improve on their own?

> Content optimization naturally targets the buyer questions where a brand performs worst. How much do those cohorts improve with no intervention at all?

- Headline result: +19pp apparent win-rate gain from pure mean reversion, with zero real change.
- Canonical: https://knitknot.ai/experiments/003-selection-bias/
- Lab notebook index: https://knitknot.ai/experiments/
- Publisher: KnitKnot, the AI Competitive Positioning platform (https://knitknot.ai)

---

**Question.** Content optimization naturally targets the buyer questions where a brand performs
worst. Badly-performing samples of a noisy metric improve on re-measurement with no
intervention at all — regression to the mean. How big is that effect here?

**Why it matters.** A random-cohort placebo ([experiment 002](/experiments/002-placebo-noise-floor/))
is centered on zero, so it can't see this bias. But no real optimization program picks cohorts
at random — it picks losers. If losers rebound on their own, every before/after comparison in
this category is upward-biased by construction, and "we improved your worst questions" is the
single easiest false claim to make.

## Method

Same null window as experiment 002 (two runs, ten days apart, nothing shipped). Instead of
random cohorts, draw 15-question cohorts from the **worst-performing** pool at baseline
(bottom tier by win rate, or by visibility), run the same production estimator, and compare
against random-cohort and best-performing-cohort draws. 200 cohorts per condition.

## Results

| Cohort selection | Mean "effect" on the null window |
|---|---|
| Worst by win rate | **+19.1pp** |
| Worst by visibility | **+17.7pp** |
| Random | −1.1pp |
| Best by win rate | −13.7pp |

Textbook mean reversion, in both directions: cohorts selected for doing badly "gain" ~18–19
points with zero real change; cohorts selected for doing well "decline" ~14.

## The nuance: what our product actually selects on

Our production pipeline selects treated questions by **issue fingerprint** (which feature,
competitor, or topic a finding is about), not directly by performance rank. We replayed the
real historical treated cohorts through the same null window: they averaged **+2.75pp** — near
the random floor, far below the +19pp worst-selection bound. Performance selection enters
indirectly (issues are minted where the brand loses), so the realistic bias sits between the
floor and the bound, and we can measure it per cohort.

## What we changed because of this

- Any claimed effect on performance-selected cohorts must beat the **measured mean-reversion
  baseline for that selection rule** — not zero.
- Sales and reporting language never uses "your worst questions improved" as evidence on its
  own; that outcome is the default expectation under no intervention.
- The selection-aware placebo joins the random placebo as a standing harness, so the bias is
  re-measured rather than remembered.

## Takeaway for buyers of any tool in this category

Ask your vendor two questions: *what does your estimator report on a window where nothing
shipped?* and *how much do your targeted cohorts improve without intervention?* If they can't
answer with numbers, their case studies are — statistically speaking — indistinguishable from
weather.