We ranked a customer first when it was tenth

A number that was too good

Until late July, the visibility rail on one customer’s home dashboard said they ranked first of 56. The customer sells AI quality assurance software to support teams, and the rail compares how often the company is named in AI answers with how often each tracked competitor is named. Zendesk sat in second place, credited with 13 answers.

Zendesk is a name almost everyone in support software knows, and 13 answers looked low for it. Counted by whether the answer text names it, Zendesk appeared in 387 of the same 584 answers. The customer’s honest rank was tenth.

I want to explain how we got there, because the query that produced first place did not look broken. Each half of it was reasonable on its own.

Two rulers

For the customer, the rail asked a question our scoring pipeline already answers for every response: is the company present in this answer at all? That is a judged field, and it is written for every answer we score.

For competitors, the rail counted rows in our claims table tagged as being about that competitor. That table was there, it was keyed by competitor, and it was cheap to count. It also measured something else entirely. Claims are what our extractor pulls out when a sentence makes an assertion worth checking, and nearly all of those assertions are about the company we’re benchmarking. A competitor named in a list of five vendors, with nothing said about it that needed validating, produced no claim row. As far as the rail knew, it wasn’t there.

So one side was measured by presence and the other by a byproduct of extraction. That asymmetry was enough to flip the ranking.

What made it easy to miss is that we had no stored record of which competitors each answer named. Scoring computes a per-competitor breakdown, but only the total count reached the table we aggregate from. The claims table looked like the closest substitute, and it was the wrong one.

One ruler

The fix was to ask the same question of every entity: does the answer text name it? We already had the list of names and spellings each company goes by. We ran that same matcher over every answer, once for the customer and once for each competitor. For the customer the result agrees with the judged presence field by construction, because scoring decides presence with the same match.

That meant rescanning answer text instead of reading a stored column. It costs nothing in model calls and is exact for every answer we have ever captured, so there was nothing to backfill. Over 584 answers and 56 entities it runs in about 0.7 seconds, which was fine behind a five-minute cache.

The same mistake, two months later

In September we found the same kind of error on the same dashboard, in a different shape. We fixed it on September 18.

The chart showed citation share inside head-to-head answers, the prompts that ask an engine to compare the customer with a named competitor. We computed every entity’s share over the whole set of those answers. That sounds fair until you notice how the set is built. A head-to-head prompt names the customer by construction, so all 550 of its head-to-head answers were about the customer. Those same answers split across eight rivals, about 69 each. Measured over the shared pool, the customer looked roughly eight times more cited than any rival, for a reason that had nothing to do with how engines treat it.

We changed the chart so each rival is measured only inside its own matchups, the answers whose verdict names that rival. The customer’s line is gone from that chart entirely, because its number over the whole pool can’t be compared with any single rival’s. A rival with no matchup in a given week gets no value, rather than a zero. A week that measured nothing draws a gap. Those are different facts from “measured, and never cited,” and the chart now says so.

The visibility chart beside it, built from discovery prompts, stayed as it was. A discovery prompt names nobody, so a shared denominator is honest there.

What I’m unsure about

Text presence is a blunt signal. Being named in a list is not the same as being recommended, and a company can be named in an answer that argues against buying it. The judged fields we use elsewhere carry more information than a name match does.

I still think the blunt, symmetric signal was the right choice for a ranking. A precise signal on one side and a crude one on the other produced a confident wrong answer. This one happened to flatter the customer, which made it more dangerous. Nobody reports a bug that tells them they’re winning.

First of 56 would have made a good slide, and so would an eightfold citation lead. Neither was true. A team that planned its content around either number would have been planning around our query.

What we check now

Both bugs came from comparing two sides that went through different code. When we build a comparison between a customer and its competitors, the first thing we look at is whether both sides are counted by the same function over the same set of answers. If they aren’t, the comparison isn’t ready to show.