Question. Feed the identical AI answer to our scoring pipeline repeatedly, with everything pinned (same model, same context, caching disabled). Do we get the same grades?

Why it matters. Every metric we show — visibility, win rate, claim accuracy — is produced by an LLM-judge pipeline. If the judge disagrees with itself, that disagreement lands inside customer metrics before the AI engines contribute any real change. A scorer that can’t agree with itself measures nothing.

Method

• Fixed set of real AI answers from a production workspace (anonymized), stratified across all coverage levels, all four engines, and stored win/loss/tie outcomes.
• Each answer scored 5 times per stage with the production judge model, temperature and context pinned, cache disabled, every upstream input frozen so each stage’s number is its own noise, not inherited noise.
• Measured: identical-across-5-passes rate, pairwise agreement, and count spread for list-producing stages.

Baseline findings (before fixes)

• Coverage classification: 16.5% pairwise disagreement; reliable at the extremes (clearly-featured / clearly-absent), soft in the middle grades.
• Claim extraction was the worst offender: identical text yielded 4, 2, 4, 1, 3 extracted claims across passes — the extractor found substantially the same content but sliced it at different granularity. This single mechanism explained most downstream count wobble.
• One list stage had a blow-up mode: on one answer it returned the entire competitor roster (55 names) on 3 of 5 passes, then 1 and 0.
• The misrepresentation verdict itself was stable: with claims pinned, zero false misrepresentation alarms in 30 repeated passes. Flips were confined to accurate ↔ unverifiable, never into “inaccurate”.

Fixes, each measured

• Pinning temperature to 0 solved coverage outright (100% identical) and helped most stages — but left claim extraction essentially unchanged. Sampling noise and output-contract noise are different diseases; temperature only treats the first.
• Segment-bound extraction: at most one claim per answer segment, enforced in parsing, so claim count is a function of the answer’s structure, not the model’s mood. Median count spread went 4 → 1.
• Bounded list schema + response-presence guard killed the roster blow-up (max spread 55 → 1).
• Closed-vocabulary keys + majority voting stabilized feature detection (0.54 → 0.79+ agreement).

Negative results (published on purpose)

• Worked boundary examples — the standard prompt-engineering fix — made feature detection worse, twice.
• A larger judge model near-emptied one stage’s output; rejected.
• Majority voting on open-ended extraction traded recall for stability at 3× cost; rejected (voting suits closed vocabularies, not open extraction).

The general lesson: measure each technique per stage. The playbook applied blind makes things worse about as often as better.

Where it stands

The customer-facing head-to-head verdict is the most stable stage measured: 91% pass-pair agreement, with disagreement confined to answers that are genuinely hedged. Remaining per-run scorer noise on portfolio win rate is low single digits of a percentage point over a full run — small relative to engine-side variance, and known rather than assumed.

Repeatability is enforced going forward: a committed harness re-runs these measurements on pinned fixtures, so a prompt or model change that destabilizes a scorer is caught before it ships.