# 002 — The placebo backtest: what does our estimator report when nothing happened?

> Take a time window in which a customer shipped no changes, pretend a change shipped, and run our own impact estimator. What "effects" does it report?

- Headline result: 95% null interval ≈ ±23pp win-rate on 15-prompt cohorts — the published noise floor.
- Canonical: https://knitknot.ai/experiments/002-placebo-noise-floor/
- Lab notebook index: https://knitknot.ai/experiments/
- Publisher: KnitKnot, the AI Competitive Positioning platform (https://knitknot.ai)

---

**Question.** Take a time window in which a customer shipped *no* changes. Pretend a change
shipped, and run our own impact estimator — the exact production code path — over random
cohorts of buyer questions. What "effects" does it report?

**Why it matters.** Whatever the estimator reports on a null window is pure noise. That
distribution is the *noise floor*: no per-fix impact number smaller than it deserves to be
published. Most tools in this category report before/after deltas with no such floor; the
independent literature (bootstrap confidence intervals of 3–6pp on citation share; run-to-run
answer overlap sometimes below 1-in-100) says the floor is large.

## Method

- A production workspace (anonymized) with a **stable question library** across two benchmark
  runs ten days apart, and verifiably zero shipped changes in between.
- 200 random cohorts of 15 questions each ("pretend this cohort was treated"), each run through
  the production difference-in-differences path: treated-cohort delta minus untouched-holdout
  delta.
- Gate: on a null window, the mean effect must be statistically indistinguishable from zero.

## Results

| Metric | Mean null effect | 95% null interval |
|---|---|---|
| Win rate | −1.1pp | **[−24.9, +22.2]pp** |
| Visibility | −0.1pp | [−19.8, +19.2]pp |

The gate passes (no *systematic* bias) — but the interval is the story: **a 15-question cohort
can swing ±23 points of win rate between two runs with zero real change.** Roughly one in five
null cohorts shows an apparent effect of 10 points or more.

## What we changed because of this

- **We do not publish per-fix effect sizes at this cohort size.** Every one of our own nine
  historical per-fix impact numbers fell inside the null band; they were retired.
- The placebo backtest is now a **standing build gate**: any change to the estimator must pass
  it on a null window before shipping.
- Proof of impact is reported as **receipts** (the AI answer that changed, citing the shipped
  page, with fetch dates) and **sustained multi-run trend**, with the noise floor printed next
  to any aggregate movement.

## Honest caveats

- The null window is assumed treatment-free because no tracked change shipped in it; customer
  changes made outside the platform aren't logged and could contaminate it. If they did, the
  true floor is *lower* than measured — which would not change the conclusion's direction.
- One workspace, one run pair. The floor will be re-estimated per customer as clean run pairs
  accumulate; nothing about the method is workspace-specific.