The citation that belonged to the next paragraph

Three sources that could not exist

On August 23 my co-founder Max opened a Gemini answer in our own KnitKnot workspace. Our pipeline had extracted several claims from it about our product. Three of those claims listed their source as trackmyvisibility.com, tryprofound.com and lemniscategrowth.com.

That should have been impossible. At the time no third party had written about KnitKnot. Any sentence Gemini wrote about our product had to come from a page we published, so every one of those attributions was wrong.

The whole promise of a claim ledger is that you can click a bad sentence and land on the page that taught the model to say it. If the link points at the wrong page, the customer goes off to argue with a company that never said anything about them.

What we thought the number meant

When we capture a Gemini answer through Bright Data, each cited source comes back with a field called position. We read it as a character offset into the answer text. That reading was reasonable. The values were integers, they fell inside the length of the answer, and they tended to land near the sentences they belonged to. So we bound each claim to the nearest position after the end of the claim.

We were confident enough to publish it. Earlier in August we wrote about our attribution ladder and listed “character offset” as the most precise rung, the one where the engine stamps the exact position of the citing sentence. This post is the correction.

When we pulled raw rows and lined them up against the text, the field turned out to mean something narrower. It is the start of the first segment that cites a given URL, counted in Bright Data’s plain-text version of the answer, and it appears once per URL. That has three consequences. It drifts away from the formatted answer by every bold marker, heading and bullet. A page cited twice only gets one position. And because it marks where a cited segment starts, “the nearest position after the claim” is usually the start of the next paragraph.

That last one did the damage. Gemini writes a comparison as alternating bullets: a paragraph about us, then a paragraph about the rival. The nearest citation after our paragraph was the one opening the rival’s paragraph. Our sentences inherited the competitor’s source, bullet after bullet. On our own workspace, 127 of 291 Gemini claims had picked up a third-party URL this way.

The same mistake with a different engine

ChatGPT had a version of the same problem. Its answers carry footnotes like [5], and we mapped the fifth footnote to the fifth source in the captured list. But that list is every search result the engine consulted, often more than thirty pages, and the footnote numbers count citation pills on the page. A pill that re-cites an earlier page, or folds several pages into one “+2” chip, shifts every number after it. In one answer the footnote on KnitKnot’s own pitch resolved to gauge.ai, an unrelated market-research firm. Of 257 footnote-bound claims about KnitKnot, 138 pointed at someone else.

Both bugs came from one habit. We treated a number in the capture as if it described what a reader sees, without checking it against what a reader sees.

Reading the marks on the page

So we stopped inferring from metadata and started reading the citation marks the answer shows. Gemini puts a short site-name label on its own line under every grounded paragraph. The newer ChatGPT interface puts a run of favicons at the end of the paragraph. The older one uses bracketed footnotes, which we now resolve through the order of citation pills we save from the page when we capture it. Across 8,903 Gemini labels in 1,313 answers, the label matched the cited source’s name 99.8% of the time.

The binding rule is plain. A claim takes the mark that closes its own paragraph. If another line of text sits between the claim and the mark, the mark belongs to a later paragraph and the claim gets nothing from it. A paragraph with no mark was not cited, so its claims carry no source.

We added one more check, because a comparative paragraph often ends with a single mark covering both halves. Gemini links each source with a fragment of the passage it quoted. If that passage is about the rival and never names the subject, the rival’s page does not get credit for a sentence about the subject.

What we gave up

The honest version costs coverage. ChatGPT answers we captured before we started saving the pill order have no map, so only their first footnote is knowable. The rest now carry no source. When we replayed the new rule over the answers we had stored, ChatGPT claims with a source fell from about 15,700 to 5,600 across all workspaces. On our own workspace, sourced claims went from 622 to 486, while claims credited to a third party fell from 282 to 43.

Most of those old footnote links were wrong, and I’m comfortable losing them. Some were probably right. We can’t tell which without re-fetching the original captures. I also don’t have a clean answer for Perplexity answers that only list sources in a footer. We leave those unbound on purpose.

A few of the third-party links that survived are interesting in their own right. They point at namesakes, like a Framer template and a knitting shop. That is identity confusion inside the answer itself, and the customer should see it.

Where this leaves us

I think a blank source is more useful than a plausible one. A blank tells the customer we couldn’t trace that sentence. A competitor’s URL on a sentence about their own product tells them something false, and they have no way to know it.

We now read the marks the reader sees, and when those marks are missing, the claim stays unsourced.