# 001 — Does our scorer agree with itself? > Feed the identical AI answer to our scoring pipeline repeatedly, with everything pinned. Do we get the same grades? - Headline result: Baseline pairwise disagreement up to 16.5% on some stages; after fixes, the customer-facing verdict stage reaches 91% pass-pair agreement, with residual disagreement confined to genuinely hedged answers. - Canonical: https://knitknot.ai/experiments/001-scorer-repeatability/ - Lab notebook index: https://knitknot.ai/experiments/ - Publisher: KnitKnot, the AI Competitive Positioning platform (https://knitknot.ai) --- **Question.** Feed the identical AI answer to our scoring pipeline repeatedly, with everything pinned (same model, same context, caching disabled). Do we get the same grades? **Why it matters.** Every metric we show — visibility, win rate, claim accuracy — is produced by an LLM-judge pipeline. If the judge disagrees with itself, that disagreement lands inside customer metrics *before* the AI engines contribute any real change. A scorer that can't agree with itself measures nothing. ## Method - Fixed set of real AI answers from a production workspace (anonymized), stratified across all coverage levels, all four engines, and stored win/loss/tie outcomes. - Each answer scored **5 times per stage** with the production judge model, temperature and context pinned, cache disabled, every upstream input frozen so each stage's number is its own noise, not inherited noise. - Measured: identical-across-5-passes rate, pairwise agreement, and count spread for list-producing stages. ## Baseline findings (before fixes) - **Coverage classification**: 16.5% pairwise disagreement; reliable at the extremes (clearly-featured / clearly-absent), soft in the middle grades. - **Claim extraction** was the worst offender: identical text yielded 4, 2, 4, 1, 3 extracted claims across passes — the extractor found substantially the same content but *sliced it at different granularity*. This single mechanism explained most downstream count wobble. - **One list stage had a blow-up mode**: on one answer it returned the entire competitor roster (55 names) on 3 of 5 passes, then 1 and 0. - **The misrepresentation verdict itself was stable**: with claims pinned, zero false misrepresentation alarms in 30 repeated passes. Flips were confined to accurate ↔ unverifiable, never into "inaccurate". ## Fixes, each measured - **Pinning temperature to 0 solved coverage outright** (100% identical) and helped most stages — but left claim extraction essentially unchanged. Sampling noise and output-contract noise are different diseases; temperature only treats the first. - **Segment-bound extraction**: at most one claim per answer segment, enforced in parsing, so claim count is a function of the answer's structure, not the model's mood. Median count spread went 4 → 1. - **Bounded list schema + response-presence guard** killed the roster blow-up (max spread 55 → 1). - **Closed-vocabulary keys + majority voting** stabilized feature detection (0.54 → 0.79+ agreement). ## Negative results (published on purpose) - Worked boundary examples — the standard prompt-engineering fix — made feature detection *worse*, twice. - A larger judge model near-emptied one stage's output; rejected. - Majority voting on open-ended extraction traded recall for stability at 3× cost; rejected (voting suits closed vocabularies, not open extraction). The general lesson: measure each technique per stage. The playbook applied blind makes things worse about as often as better. ## Where it stands The customer-facing head-to-head verdict is the most stable stage measured: **91% pass-pair agreement**, with disagreement confined to answers that are genuinely hedged. Remaining per-run scorer noise on portfolio win rate is low single digits of a percentage point over a full run — small relative to engine-side variance, and known rather than assumed. Repeatability is enforced going forward: a committed harness re-runs these measurements on pinned fixtures, so a prompt or model change that destabilizes a scorer is caught before it ships.