# KnitKnot Measurement Lab

> How we measure what AI says about brands — and how wrong we could be. Our open lab notebook: every error bar we publish, every claim we make about our own accuracy, traces back to an experiment written up here — method, numbers, and what we changed because of it.

## Experiments

- 001 — Does our scorer agree with itself?
  Question: Feed the identical AI answer to our scoring pipeline repeatedly, with everything pinned. Do we get the same grades?
  Headline result: Baseline pairwise disagreement up to 16.5% on some stages; after fixes, the customer-facing verdict stage reaches 91% pass-pair agreement, with residual disagreement confined to genuinely hedged answers.
  Full write-up: https://knitknot.ai/experiments/001-scorer-repeatability/ (markdown mirror: https://knitknot.ai/experiments/001-scorer-repeatability.md)
- 002 — The placebo backtest: what does our estimator report when nothing happened?
  Question: Take a time window in which a customer shipped no changes, pretend a change shipped, and run our own impact estimator. What "effects" does it report?
  Headline result: 95% null interval ≈ ±23pp win-rate on 15-prompt cohorts — the published noise floor.
  Full write-up: https://knitknot.ai/experiments/002-placebo-noise-floor/ (markdown mirror: https://knitknot.ai/experiments/002-placebo-noise-floor.md)
- 003 — Regression to the mean: how much do "worst" cohorts improve on their own?
  Question: Content optimization naturally targets the buyer questions where a brand performs worst. How much do those cohorts improve with no intervention at all?
  Headline result: +19pp apparent win-rate gain from pure mean reversion, with zero real change.
  Full write-up: https://knitknot.ai/experiments/003-selection-bias/ (markdown mirror: https://knitknot.ai/experiments/003-selection-bias.md)
- 004 — CUPED: using each question's own history to shrink the noise floor
  Question: The null interval from experiment 002 is ±23pp — too wide to read small cohorts. Can each question's baseline value, used as a covariate, buy the floor down?
  Headline result: 27% narrower error bars for free — and the optimal adjustment coefficient turns out to be the mean-reversion slope from experiment 003.
  Full write-up: https://knitknot.ai/experiments/004-cuped-variance-reduction/ (markdown mirror: https://knitknot.ai/experiments/004-cuped-variance-reduction.md)

## Ground rules

- Customer workspaces are anonymized.
- Negative results are published: several intuitive fixes (worked examples, majority voting on open extraction, bigger judge models) measurably did not help or made things worse, and the write-ups say so.
- When an experiment invalidates something we previously claimed, the claim is retired.

Methodology summary: https://knitknot.ai/methodology/