The report that started a fight

Two weeks ago we walked Drata’s head of growth through a benchmark report we’d generated for them. Drata is one of the leaders in the compliance automation category. They have a real perspective on how they get represented online, and a real opinion about what fair benchmarking looks like.

About a third of the way in, he stopped me.

“It looks like you engineered them to win this.”

He was pointing at the prompts. The questions we feed into ChatGPT and Claude to test how AI represents a company. And honestly, he had a point. We had been writing prompts designed to surface the worst representation possible. Things like “What are the hidden problems with Drata?” and “Why do people switch from Drata to Vanta?” We thought we were doing companies a favor by showing them the absolute worst case.

The customer saw it differently. If the test looks rigged, the results don’t matter. You can’t show someone their blind spots if they don’t trust your eyes.

Why we wrote them that way

I want to explain why we did it, because the instinct wasn’t wrong.

When you build a benchmark, you want it to show areas for improvement. A report card that says “you’re doing great everywhere” is useless. Nobody learns from it, nobody acts on it, nobody shares it with their team. So we leaned into antagonistic prompts. Stress-test the brand. Find where AI says the worst things. Show the bleeding.

We had seven categories of prompts at that point. Head-to-head comparisons. Brand perception probes. Negative sentiment. Each one was designed to find a different kind of weakness. And the prompts worked in the sense that they found real problems. AI was saying inaccurate things about companies, and our prompts caught it.

But we had started to drift. The prompts were adversarial by construction, not by accident. We were generating questions no real buyer would ever type. “Compare Drata and Vanta on documentation aesthetics.” Nobody has ever asked ChatGPT that. We were measuring something, but it wasn’t something anyone cared about.

The prompt is half the measurement. If you ask a loaded question, you get a loaded answer. We knew this in principle but we hadn’t applied it to our own product.

The two questions we couldn’t answer

After the Drata call we sat down and asked two questions we’d been avoiding.

First: is this prompt something a real buyer would actually type into ChatGPT? For a lot of our prompts, the honest answer was no. We were stress-testing in a way that felt satisfying to us but didn’t reflect how buyers actually research software.

Second: would a reasonable person look at this prompt and say it’s fair? Again, for many of them, no. We were leading the witness toward the AI, and anyone who read the prompt itself could see it.

We realized we had a trust problem, not a coverage problem. More prompts wouldn’t fix it. More categories wouldn’t fix it. The prompts themselves had to be defensible.

Grounding every prompt in a real search

So we rebuilt prompt generation around a single constraint: every prompt has to be tied to a real Google query that real buyers type.

We built what we call a grounding service. For each company and its competitors, we pull keyword data from DataForSEO. Not just “X vs Y” queries, but the long tail: “Drata vs Vanta for SOC 2,” “compliance automation for startups,” “best GRC tools for enterprise.” Three endpoints per seed, suggestions and ideas and related keywords, all fanned out and deduplicated.

Each prompt gets assigned a grounding tier:

• Direct if it maps to a query like “Drata vs Vanta” that has measurable monthly search volume. This is the strongest signal. Real buyers are asking this exact question right now.
• Category if direct queries are thin but category-level queries exist (“compliance automation tools”). Tests whether AI surfaces you when buyers ask about the category.
• Synthesized if there’s no measurable volume at all. GPT-generated, labeled as such. Common for brand-new companies. Your AI presence baseline starts here.

Every prompt in the system now carries its grounding source. Search volume. CPC. Competition index. The customer can see exactly why we asked that question and how many real buyers are asking the same thing.

Finding the actual buyers

The other thing we got wrong was assuming we knew who the buyer was.

Our prompts used to frame around hardcoded personas. “I’m a VP of Engineering evaluating…” or “As an IT director, help me decide…” These were guesses. Reasonable guesses, but guesses. And they made the prompts feel artificial in a different way.

We replaced them with research-discovered personas. When we start researching a company, we now discover six to eight actual buyer profiles based on the company’s positioning, features, and product lines. A CISO at a 5,000-person enterprise has different priorities and constraints than a Compliance Manager at a 200-person startup. The prompts reflect that specificity. Not “I’m evaluating compliance tools” but “I’m a CISO at a Series C company, I need to consolidate audit tools before our next board review, and I’m comparing Drata and Vanta.”

This matters because the persona changes the question. A CISO asks about vendor consolidation and audit readiness. A DevOps lead asks about API integrations and deployment friction. Same product category, entirely different prompts, entirely different AI responses.

Reading what competitors are actually publishing

The last piece is the newest.

We noticed that AI’s misrepresentations often came from competitor content. Not because competitors were lying, but because they were publishing more, and about the specific features where AI was getting things wrong. AI was reading their blog posts and help docs and repeating their framing back to buyers.

So we built an article crawl pipeline. After every benchmark run, we fetch the top 10 competitor URLs that AI cited in feature losses. We extract the actual article text. Then we use GPT-4o to pull 3-5 verified quotes from each article, the specific sentences that are training AI to view the company a certain way. Every quote is verified against the original text. If the substring doesn’t match, it gets dropped.

This changed the report from “AI said this about you” to “here’s the article that taught AI to say that.” It’s one thing to know you’re losing. It’s another thing to see the exact paragraph your competitor published that made it happen.

What good benchmarking looks like

We still believe you need to see the worst case. A report card that only highlights strengths is a press release, not a diagnosis. But the worst case has to come from reality, not from us engineering it.

Every prompt we generate now is symmetric (same structure regardless of who “should” win), grounded in a real Google query with monthly volume, and framed from the perspective of a real buyer persona we discovered through research.

We expect this to be the methodology line that comes up in every sales conversation: every prompt is symmetric and grounded in a real Google query with monthly search volume. Read it yourself.

That’s what good benchmarking looks like. Thanks to the Drata team for the push.

What’s next

The current grounding source is Google search volume via DataForSEO. The next layer we’re working on is Reddit search behavior. What buyers ask each other in unmoderated forums often diverges from what they type into Google. When we add Reddit grounding, we’ll publish a follow-up here.

If you want a benchmark for your category, symmetric and grounded in real queries, book a slot.