Question. Experiment 002 measured our per-cohort noise floor: a 95% null interval roughly ±23pp wide at 15-question cohorts. Variance-reduction techniques from large-scale A/B testing (CUPED — Controlled-experiment Using Pre-Experiment Data) claim you can shrink error bars by regressing out each unit’s pre-period value. Does that work at our scale, and by how much?

Why it matters. Every point of variance we remove is detection power we don’t have to buy with more questions or more runs. And the standard estimator everyone in this category uses — subtract baseline from measurement — is itself a special case of CUPED with the adjustment coefficient θ pinned to 1. If the optimal θ isn’t 1, plain before/after subtraction is not just noisy, it’s the wrong member of its own estimator family.

Method

Same null window as experiments 002/003 (two runs, ten days apart, nothing shipped; 286 shared questions). For each question, X = its baseline-run value, Y = its measurement-run value. Fit θ = cov(X,Y)/var(X) once across all shared questions, then score each of the 200 null cohorts with the CUPED-adjusted effect: mean(Y − θX) over the cohort minus the same over the holdout. Compare the resulting null distribution’s width against the plain θ=1 estimator on identical cohorts, so the only difference is θ.

Results

Metric	θ	Plain 95% null interval	CUPED 95% null interval	Shrinkage
Win rate	0.52	[−24.9, +22.2]pp (47.1pp wide)	[−18.0, +16.1]pp (34.1pp wide)	−27.6%
Visibility	0.13	[−19.8, +19.2]pp (39.0pp wide)	[−13.0, +15.4]pp (28.4pp wide)	−27.1%

The connective tissue: θ is the mean-reversion slope

θ_win = 0.52 means a question keeps only about half of its baseline deviation from run to run — which is exactly the regression to the mean that experiment 003 measured as a +19pp bias on worst-selected cohorts. Plain subtraction (θ=1) assumes deviations persist fully, so it over-subtracts the baseline and adds variance relative to the optimum. The same θ<1 that shrinks the error bars also partially deflates the mean-reversion bias that loser-selected cohorts carry. One measured constant, two of our known failure modes.

What we changed because of this

• The CUPED-adjusted effect is being added to our measurement estimator as an additive field — frozen θ per metric, stamped alongside the raw deltas, never refit per customer (a refit on small cohorts is noisier than the bias it removes).
• Permutation p-values are now computed against the placebo null distribution we already generate, instead of assuming any parametric shape.

The honest limit

27% narrower is material — it is not a rescue. At 15-question cohorts the CUPED floor is still roughly ±17pp. Small-cohort per-play effect sizes remain unreadable, which is why our reporting leads with census receipts (verbatim answers, citations of the shipped page) and treats deltas as confirmation, not proof.

Takeaway for buyers of any tool in this category

If a vendor reports before/after deltas, ask what their adjustment coefficient is. “We subtract the baseline” means θ=1 — and on our data the correct value is about half that. A vendor who has never measured θ is running the noisiest member of the only estimator family they use.