KnitKnot
You are reading the agent-optimized layer of this page: the literal markdown we serve to AI crawlers and assistants, shipped in the page source of every visit. Making sure AI reads the right facts about a company is literally what KnitKnot does.

# Runs and scoring

What happens during a benchmark run — real AI responses, claim-level scoring, and evidence you can audit.

A run is one full benchmark: your active prompt library executed across ChatGPT, Claude, Perplexity, and Gemini, every response scored, and the results rolled into your report. Runs typically take a few hours end to end, and you can watch progress live in the console.

## Real responses, not simulations

Each prompt is executed against its selected supported engines. KnitKnot stores the returned answer, citations, engine, and model version as a controlled benchmark record. Consumer chat sessions can differ because of personalization, geography, model version, and execution path.

## Claim-level scoring

Responses aren't graded with a single thumbs-up. A scoring pipeline breaks each response down:

  • - **Presence** — how central were you to the answer, from primary subject to absent.
  • - **Claims** — every substantive statement about you is extracted with its verbatim quote, marked accurate, inaccurate, or outdated, and where the response cites sources, tied to the URL that fed it.
  • - **Head-to-head verdict** — for comparison prompts, who the AI actually picked, per feature and overall. Verdicts are conservative: hedged answers count as ties, not wins.
  • - **Sentiment and framing** — how favorably you're characterized when you appear.

Scoring uses an AI judge for the semantic work — deciding whether a paragraph is a recommendation is not a keyword-matching problem — and the judge itself is continuously calibrated against a hand-labeled gold set.

## Written once, read everywhere

Scoring results are written down once per response, and every surface — report, dashboard, drill-downs, trend charts — reads the same records. The headline number and the list of evaluations behind it can't disagree, because they're the same data.

## Auditable by design

Every aggregate drills to its evidence: score → evaluations → response text → claim → cited source. When your report says you lost a comparison, you can read the exact sentences where the AI picked the other vendor. If a scored result looks wrong, you can re-score an individual response from the console.

## Cadence

Benchmarks re-run on a schedule so your score trends without manual effort, and you can trigger a run any time — most usefully after shipping a fix, to check whether the AI's answers moved. See [Issues and playbooks](/docs/issues-and-playbooks/) for how that loop closes.

Raw mirror of this content: https://knitknot.ai/docs/runs-and-scoring.md. Site-wide summary: /llms.txt · full content: /llms-full.txt

Docs navigation
Docs › Core concepts

Runs and scoring

What happens during a benchmark run — real AI responses, claim-level scoring, and evidence you can audit.

Updated

A run is one full benchmark: your active prompt library executed across ChatGPT, Claude, Perplexity, and Gemini, every response scored, and the results rolled into your report. Runs typically take a few hours end to end, and you can watch progress live in the console.

Real responses, not simulations

Each prompt is executed against its selected supported engines. KnitKnot stores the returned answer, citations, engine, and model version as a controlled benchmark record. Consumer chat sessions can differ because of personalization, geography, model version, and execution path.

Claim-level scoring

Responses aren’t graded with a single thumbs-up. A scoring pipeline breaks each response down:

  • Presence — how central were you to the answer, from primary subject to absent.
  • Claims — every substantive statement about you is extracted with its verbatim quote, marked accurate, inaccurate, or outdated, and where the response cites sources, tied to the URL that fed it.
  • Head-to-head verdict — for comparison prompts, who the AI actually picked, per feature and overall. Verdicts are conservative: hedged answers count as ties, not wins.
  • Sentiment and framing — how favorably you’re characterized when you appear.

Scoring uses an AI judge for the semantic work — deciding whether a paragraph is a recommendation is not a keyword-matching problem — and the judge itself is continuously calibrated against a hand-labeled gold set.

Written once, read everywhere

Scoring results are written down once per response, and every surface — report, dashboard, drill-downs, trend charts — reads the same records. The headline number and the list of evaluations behind it can’t disagree, because they’re the same data.

Auditable by design

Every aggregate drills to its evidence: score → evaluations → response text → claim → cited source. When your report says you lost a comparison, you can read the exact sentences where the AI picked the other vendor. If a scored result looks wrong, you can re-score an individual response from the console.

Cadence

Benchmarks re-run on a schedule so your score trends without manual effort, and you can trigger a run any time — most usefully after shipping a fix, to check whether the AI’s answers moved. See Issues and playbooks for how that loop closes.