KnitKnot
You are reading the agent-optimized layer of this page: the literal markdown we serve to AI crawlers and assistants, shipped in the page source of every visit. Making sure AI reads the right facts about a company is literally what KnitKnot does.

# KnitKnot Measurement Lab

How we measure what AI says about brands — and how wrong we could be. Our open lab notebook: every error bar we publish, every claim we make about our own accuracy, traces back to an experiment written up here — method, numbers, and what we changed because of it.

## Experiments

## Ground rules

  • - Customer workspaces are anonymized.
  • - Negative results are published: several intuitive fixes (worked examples, majority voting on open extraction, bigger judge models) measurably did not help or made things worse, and the write-ups say so.
  • - When an experiment invalidates something we previously claimed, the claim is retired.

Methodology summary: https://knitknot.ai/methodology/

Raw mirror of this content: https://knitknot.ai/experiments.md. Site-wide summary: /llms.txt · full content: /llms-full.txt

KnitKnot index

Measurement lab

Our open lab notebook. Every error bar we publish, and every claim we make about our own accuracy, traces back to an experiment written up here — method, numbers, and what we changed because of it.

  1. Experiment 001

    Does our scorer agree with itself?

    Feed the identical AI answer to our scoring pipeline repeatedly, with everything pinned. Do we get the same grades?

    Result: Baseline pairwise disagreement up to 16.5% on some stages; after fixes, the customer-facing verdict stage reaches 91% pass-pair agreement, with residual disagreement confined to genuinely hedged answers.

  2. Experiment 002

    The placebo backtest: what does our estimator report when nothing happened?

    Take a time window in which a customer shipped no changes, pretend a change shipped, and run our own impact estimator. What "effects" does it report?

    Result: 95% null interval ≈ ±23pp win-rate on 15-prompt cohorts — the published noise floor.

  3. Experiment 003

    Regression to the mean: how much do "worst" cohorts improve on their own?

    Content optimization naturally targets the buyer questions where a brand performs worst. How much do those cohorts improve with no intervention at all?

    Result: +19pp apparent win-rate gain from pure mean reversion, with zero real change.

  4. Experiment 004

    CUPED: using each question's own history to shrink the noise floor

    The null interval from experiment 002 is ±23pp — too wide to read small cohorts. Can each question's baseline value, used as a covariate, buy the floor down?

    Result: 27% narrower error bars for free — and the optimal adjustment coefficient turns out to be the mean-reversion slope from experiment 003.

Ground rules

How the notebook is kept honest.

Anonymized

Customer workspaces are anonymized in every write-up.

Negative results are published

Several intuitive fixes — worked examples, majority voting on open extraction, bigger judge models — measurably did not help or made things worse, and the write-ups say so.

Invalidated claims are retired

When an experiment invalidates something we previously claimed, the claim is retired.

For the plain-language summary of what these experiments mean for how we report, see our methodology.