KnitKnot

# Benchmark questions

Where benchmark questions come from, how the budget is split across topics by real buyer demand, and what a given library size can honestly claim.

Your question library is the set of buyer-style questions every benchmark asks. It's the measuring stick, so it is built to be realistic, stable, and yours to curate.

## Grounded in real demand

Questions aren't synthetic templates. Generation starts from real Google search queries in your category, with monthly search volume attached, and turns them into the questions a buyer would ask an AI assistant: "best X for Y", "compare A vs B", "does A support Z". Each question records why it was generated : the keyword, capability, or buyer role behind it : so you can always trace a question back to the demand it represents.

The [ChatGPT queries observed during a benchmark](/docs/inspect-chatgpt-retrieval/) are a separate record. Their count shows captured retrieval activity, not monthly buyer search demand.

To see the topics and keywords behind that demand, use [Read search demand by topic and keyword](/docs/read-search-demand/).

The mix leans deliberately competitive: most questions put you in a buying context against named competitors, because that is where deals are won and lost. The rest are open landscape questions that measure organic visibility: whether AI brings you up without being told to.

## How the budget is split: topics first, by demand

Your keywords are grouped into **topics** : the handful of things buyers in your category actually search for. A topic is the unit of demand, because search volume is measured per topic, not per individual question. Your library size is then split across those topics in proportion to that demand.

The split is deliberately **compressed**, not strictly proportional. A topic with 750 searches a month gets roughly three times the questions of one with 10 : not seventy-five times. Head topics should lead; they shouldn't swallow the whole library and leave you blind everywhere else.

Within a topic, the allocation is spent on **depth** rather than breadth. Competitors and buyer roles are how a topic varies its questions, not separate budgets of their own. A topic big enough to fund four competitors properly carries four; a smaller one carries two at full depth instead of one shallow question about each.

## Every measured slice gets at least three questions

**A topic either earns at least three questions or it isn't measured** : and when it isn't, we say so rather than report on it anyway.

Three is the floor because one question, even when run on several engines, tests only one phrasing. If those answers come back badly, you have learned about that question, not established that AI is silent on the whole topic.

The same floor applies at reporting time. A finding is never drawn from a slice measured by fewer than three questions, so if a topic falls under the bar in a given week : never allocated, or an assistant failed mid-run : it reads as **not measured this period**. Absence of measurement is not absence from the answer, and we won't print one as the other.

The consequence is that a low-demand topic can drop out of a small library entirely. That's intentional: nine topics measured well beats thirteen measured unreliably.

**Your own capabilities are the exception.** Keyword tools have no search volume for plenty of things buyers genuinely evaluate you on. A topic that represents one of your actual product capabilities can use estimated demand; [Prompt library → Topics](https://app.knitknot.ai/prompts) marks that estimate. A keyword topic nobody searches for is weak evidence, while a documented product capability can still warrant measurement.

## What your library size can claim

More prompts can support more detailed cuts of a benchmark, but the number needed depends on the topics, competitors, engines, and successful answers in that period. A library may support a topic-level view while leaving a topic-by-competitor slice too thin to report. Check the measured sample on a result before treating that narrower slice as a finding.

## Persistent, not regenerated

The library persists across runs. Each benchmark snapshots the prompts active at that time. Two runs are most comparable when the active prompts and engines match; a library edit can change the result as well as the answers.

The library can hold archived prompts alongside the active set. When demand, competitors, or plan scope changes, review which prompts are active before interpreting the next benchmark. Past answers remain attached to their original prompts.

**Manual and starred prompts remain identifiable.** Starring marks priority; it does not activate an archived prompt. Review status in [Prompt library](https://app.knitknot.ai/prompts) to see what will run.

You can add, star, activate, and archive prompts between runs. Archived prompts stop being benchmarked but keep their history; new active prompts begin contributing from the next run.

## Per-subject libraries

Each [subject](/docs/subjects/) : your brand and each product : has its own library, generated from its own category and keywords. A benchmark for a product runs that product's questions against that product's competitors.

## Curating well

  • - **Volume beats cleverness.** A question tied to a query real buyers search monthly is worth ten hypothetical ones.
  • - **Keep the set stable.** Resist rewording questions between runs; every edit resets that question's comparability.
  • - **Prune what you'd never act on.** If a question's outcome wouldn't change what you write or fix, archive it : it's diluting the questions that matter.
  • - **Depth beats coverage.** Three questions on one comparison tell you something. One question on each of three comparisons tells you nothing, three times.
Docs navigation
Docs › Core concepts

Benchmark questions

Where benchmark questions come from, how the budget is split across topics by real buyer demand, and what a given library size can honestly claim.

Updated

Your question library is the set of buyer-style questions every benchmark asks. It’s the measuring stick, so it is built to be realistic, stable, and yours to curate.

Grounded in real demand

Questions aren’t synthetic templates. Generation starts from real Google search queries in your category, with monthly search volume attached, and turns them into the questions a buyer would ask an AI assistant: “best X for Y”, “compare A vs B”, “does A support Z”. Each question records why it was generated : the keyword, capability, or buyer role behind it : so you can always trace a question back to the demand it represents.

The ChatGPT queries observed during a benchmark are a separate record. Their count shows captured retrieval activity, not monthly buyer search demand.

To see the topics and keywords behind that demand, use Read search demand by topic and keyword.

The mix leans deliberately competitive: most questions put you in a buying context against named competitors, because that is where deals are won and lost. The rest are open landscape questions that measure organic visibility: whether AI brings you up without being told to.

How the budget is split: topics first, by demand

Your keywords are grouped into topics : the handful of things buyers in your category actually search for. A topic is the unit of demand, because search volume is measured per topic, not per individual question. Your library size is then split across those topics in proportion to that demand.

The split is deliberately compressed, not strictly proportional. A topic with 750 searches a month gets roughly three times the questions of one with 10 : not seventy-five times. Head topics should lead; they shouldn’t swallow the whole library and leave you blind everywhere else.

Within a topic, the allocation is spent on depth rather than breadth. Competitors and buyer roles are how a topic varies its questions, not separate budgets of their own. A topic big enough to fund four competitors properly carries four; a smaller one carries two at full depth instead of one shallow question about each.

Every measured slice gets at least three questions

A topic either earns at least three questions or it isn’t measured : and when it isn’t, we say so rather than report on it anyway.

Three is the floor because one question, even when run on several engines, tests only one phrasing. If those answers come back badly, you have learned about that question, not established that AI is silent on the whole topic.

The same floor applies at reporting time. A finding is never drawn from a slice measured by fewer than three questions, so if a topic falls under the bar in a given week : never allocated, or an assistant failed mid-run : it reads as not measured this period. Absence of measurement is not absence from the answer, and we won’t print one as the other.

The consequence is that a low-demand topic can drop out of a small library entirely. That’s intentional: nine topics measured well beats thirteen measured unreliably.

Your own capabilities are the exception. Keyword tools have no search volume for plenty of things buyers genuinely evaluate you on. A topic that represents one of your actual product capabilities can use estimated demand; Prompt library → Topics marks that estimate. A keyword topic nobody searches for is weak evidence, while a documented product capability can still warrant measurement.

What your library size can claim

More prompts can support more detailed cuts of a benchmark, but the number needed depends on the topics, competitors, engines, and successful answers in that period. A library may support a topic-level view while leaving a topic-by-competitor slice too thin to report. Check the measured sample on a result before treating that narrower slice as a finding.

Persistent, not regenerated

The library persists across runs. Each benchmark snapshots the prompts active at that time. Two runs are most comparable when the active prompts and engines match; a library edit can change the result as well as the answers.

The library can hold archived prompts alongside the active set. When demand, competitors, or plan scope changes, review which prompts are active before interpreting the next benchmark. Past answers remain attached to their original prompts.

Manual and starred prompts remain identifiable. Starring marks priority; it does not activate an archived prompt. Review status in Prompt library to see what will run.

You can add, star, activate, and archive prompts between runs. Archived prompts stop being benchmarked but keep their history; new active prompts begin contributing from the next run.

Per-subject libraries

Each subject : your brand and each product : has its own library, generated from its own category and keywords. A benchmark for a product runs that product’s questions against that product’s competitors.

Curating well

  • Volume beats cleverness. A question tied to a query real buyers search monthly is worth ten hypothetical ones.
  • Keep the set stable. Resist rewording questions between runs; every edit resets that question’s comparability.
  • Prune what you’d never act on. If a question’s outcome wouldn’t change what you write or fix, archive it : it’s diluting the questions that matter.
  • Depth beats coverage. Three questions on one comparison tell you something. One question on each of three comparisons tells you nothing, three times.