- Published
- Reading time
- 4 minutes
Guide
What does an AI comparison win rate mean when answers tie?
A win rate can hide two different problems: AI prefers a competitor, or it declines to choose. Keep wins, ties, and losses visible before changing your content.
Why can a good comparison record have a low win rate?
Because a tie can count in the total. Suppose you capture eight answers to buyer questions comparing your product with a rival. Four favor you, three name no winner, and one favors the rival. These are example counts, not a KnitKnot customer result.
| Outcome | Answers | What it means in this example |
|---|---|---|
| Win | 4 | AI favors you |
| Tie | 3 | AI compares the vendors but does not choose one |
| Loss | 1 | AI favors the rival |
With ties included, your win rate is 4 ÷ 8 = 50%. Among answers that picked a winner, your decided win rate is 4 ÷ 5 = 80%. Neither number is wrong. They answer different questions.
The gap between them tells you to read the tied answers. If you only saw 50%, you might assume the rival won half the comparisons. It won one of eight. If you only saw 80%, you might miss that three answers gave the buyer no clear recommendation.
What should count as a tie?
A tie is a comparison outcome where the answer does not favor either vendor overall. That can happen because the answer says the choice depends on a buyer’s needs, because each vendor wins a different criterion, or because it lacks enough evidence to decide. Those are different stories. Read the captured answer before assigning a cause.
Keep not compared separate. A question that did not compare vendors, or an answer without a usable comparison outcome, should not be quietly treated as a tie. Ask your measurement provider how it handles those answers and whether the raw response is available.
Which rate belongs in a report?
Show the wins, ties, and losses together, then label each percentage with its denominator:
- Win rate: wins ÷ (wins + ties + losses). This tells you how often you were the clear winner among measured head-to-head outcomes.
- Decided win rate: wins ÷ (wins + losses). This tells you how often you won when AI picked a winner. It is undefined when there are no wins or losses.
- Tie share: ties ÷ (wins + ties + losses). This shows how often the answer left the comparison unresolved.
In the example above, tie share is 3 ÷ 8 = 37.5%. Keep the count next to the percentage. A change from one win in two answers to two wins in three answers is too small a base to present as a settled shift in buyer perception.
Other products may define “win rate” differently. Ask for the formula rather than assuming their percentages are directly comparable. KnitKnot’s metrics reference defines both rates used in its Head-to-head table.
What do you investigate next?
Open a few ties and losses from the same competitor, buyer question, and AI engine. Look for the criterion that shaped each answer: price, feature scope, integration, security, implementation effort, or something else. Then check the statement against your approved product facts and the sources the answer actually exposes.
If the answer gets a material fact wrong, gather the exact quote and a current source before preparing a correction. If it accurately describes a limitation, the next move may be a clearer positioning page or a product decision. If it simply says “it depends,” see whether the comparison names the buyer conditions that would make your product the better fit. A blanket “we win” page would not answer that question.
After a change, rerun a comparable set of questions. Keep the prompts, engines, and outcome rules steady enough to see what changed, and read the answers again. A later improvement is an observation; it does not prove that one page caused it. For the product workflow, see Read your report and Investigate a claim.