# 004 — CUPED: using each question's own history to shrink the noise floor

> The null interval from experiment 002 is ±23pp — too wide to read small cohorts. Can each question's baseline value, used as a covariate, buy the floor down?

- Headline result: 27% narrower error bars for free — and the optimal adjustment coefficient turns out to be the mean-reversion slope from experiment 003.
- Canonical: https://knitknot.ai/experiments/004-cuped-variance-reduction/
- Lab notebook index: https://knitknot.ai/experiments/
- Publisher: KnitKnot, the AI Competitive Positioning platform (https://knitknot.ai)

---

**Question.** Experiment [002](/experiments/002-placebo-noise-floor/) measured our per-cohort
noise floor: a 95% null interval roughly ±23pp wide at 15-question cohorts. Variance-reduction
techniques from large-scale A/B testing (CUPED — Controlled-experiment Using Pre-Experiment
Data) claim you can shrink error bars by regressing out each unit's pre-period value. Does that
work at our scale, and by how much?

**Why it matters.** Every point of variance we remove is detection power we don't have to buy
with more questions or more runs. And the standard estimator everyone in this category uses —
subtract baseline from measurement — is itself a special case of CUPED with the adjustment
coefficient θ pinned to 1. If the optimal θ isn't 1, plain before/after subtraction is not just
noisy, it's the *wrong member of its own estimator family*.

## Method

Same null window as experiments 002/003 (two runs, ten days apart, nothing shipped; 286 shared
questions). For each question, X = its baseline-run value, Y = its measurement-run value. Fit
θ = cov(X,Y)/var(X) once across all shared questions, then score each of the 200 null cohorts
with the CUPED-adjusted effect: mean(Y − θX) over the cohort minus the same over the holdout.
Compare the resulting null distribution's width against the plain θ=1 estimator on identical
cohorts, so the only difference is θ.

## Results

| Metric | θ | Plain 95% null interval | CUPED 95% null interval | Shrinkage |
|---|---|---|---|---|
| Win rate | 0.52 | [−24.9, +22.2]pp (47.1pp wide) | [−18.0, +16.1]pp (34.1pp wide) | **−27.6%** |
| Visibility | 0.13 | [−19.8, +19.2]pp (39.0pp wide) | [−13.0, +15.4]pp (28.4pp wide) | **−27.1%** |

## The connective tissue: θ is the mean-reversion slope

θ_win = 0.52 means a question keeps only about half of its baseline deviation from run to run
— which is exactly the regression to the mean that experiment
[003](/experiments/003-selection-bias/) measured as a +19pp bias on worst-selected cohorts.
Plain subtraction (θ=1) assumes deviations persist fully, so it over-subtracts the baseline and
*adds* variance relative to the optimum. The same θ<1 that shrinks the error bars also
partially deflates the mean-reversion bias that loser-selected cohorts carry. One measured
constant, two of our known failure modes.

## What we changed because of this

- The CUPED-adjusted effect is being added to our measurement estimator as an additive field —
  frozen θ per metric, stamped alongside the raw deltas, never refit per customer (a refit on
  small cohorts is noisier than the bias it removes).
- Permutation p-values are now computed against the placebo null distribution we already
  generate, instead of assuming any parametric shape.

## The honest limit

27% narrower is material — it is not a rescue. At 15-question cohorts the CUPED floor is still
roughly ±17pp. Small-cohort per-play effect sizes remain unreadable, which is why our reporting
leads with census receipts (verbatim answers, citations of the shipped page) and treats deltas
as confirmation, not proof.

## Takeaway for buyers of any tool in this category

If a vendor reports before/after deltas, ask what their adjustment coefficient is. "We subtract
the baseline" means θ=1 — and on our data the correct value is about half that. A vendor who
has never measured θ is running the noisiest member of the only estimator family they use.