How Can I Detect Cherry-Picked Prompts in an AI Search Case Study?

Why is cherry-picking so easy in AI search?

Because AI answers vary a lot from run to run and from phrasing to phrasing:

• Run-to-run variation. SparkToro's study of AI recommendations found less than a 1 in 100 chance that ChatGPT or Google's AI would give the same list of brands twice for the same prompt.
• Phrasing variation. In a separate SparkToro test, 142 people wrote prompts for the same need, and their prompts were almost entirely different from each other, with an average semantic similarity of 0.081. Yet the same top brands still appeared in 55–77% of the 994 answers: natural rewording mostly keeps the same shortlist, so it takes unusual wording, such as the client's own terms or extra qualifiers, to move a brand in or out.

With that much variation, anyone who tests enough prompts will find some where the client looks great. Reporting only those is cherry-picking, whether deliberate or not.

How much can selection alone inflate results?

We simulated 100 prompts where a brand's true chance of appearing did not change at all between two test rounds, and averaged the results over many repeats of the simulation. Each prompt was run five times per round, with appearance rates varying widely by prompt, as they do in practice.

Nothing happened, yet the "top 10 prompts" show a near-tripling, and with single runs they show a jump from never to almost always. This is regression to the mean plus selection. It can make a case study overstate results without anyone intending to mislead.

What are the red flags?

• Branded prompts. "Is [client] good for X?" will mention the client whatever happens. Branded prompts inflate scores, as covered in whether brand-name prompts inflate AI visibility scores.
• Prompts written in the client's own words. A prompt that reads like the client's tagline, or lists features only the client has, makes the client the obvious answer.
• Hyper-specific prompts that no buyer would type, for example five qualifiers ending in the client's home town.
• No "before" on identical prompts. An "after" screenshot proves nothing without the same prompt tested beforehand.
• Wording that changed between rounds.
• Single runs and screenshots instead of rates across repeated runs.
• A small set with no full list, such as "we won 9 of our 10 target prompts".
• Prompts chosen after results were known, or no stated selection method.
• No controls. Platform-wide changes can lift everyone, as covered in what a control prompt is in AI search visibility measurement.
• Personalised testing. An account whose memory or instructions mention the client will favour it.
• One assistant, one mode, one location, presented as "AI search" in general.
• An undefined "visibility score" that combines mentions, citations and rank in unknown proportions.

What should you ask the vendor?

1. How many prompts did you track in total, and can we see the full list?
2. How were the prompts chosen: from customer interviews, search data, sales calls? And when were they chosen, before or after the work?
3. Were any prompts dropped or reworded during the project, and why?
4. How many runs per prompt, on which assistants, modes, locations and accounts?
5. Do you have before-and-after results on identical prompts?
6. What happened on prompts that did not improve, or got worse?
7. Did you track control prompts, and what did they do?
8. Can we see the raw answers or an export?

A vendor confident in its results will answer all eight. Vague answers to questions 1, 2 and 6 are the clearest sign of selection.

How do you retest a claim yourself?

• Rerun a sample of their prompts in a clean account or unpersonalised session, several times each, on the same assistant and mode.
• Write your own buyer-style prompts for the same need, unbranded and in natural language, and test those too. If the client appears only for the vendor's prompts, the result will not reach real buyers.
• Test competitors on the same prompts, to see whether the client stands out or everyone appears.
• Compare with a neutral topic the vendor did not work on, as a rough control.

How to choose prompts that reflect real demand is covered in how to choose the right prompts to track for AI visibility.

What does a trustworthy case study include?

• A prompt set fixed before the work, with its source and the full list, or a representative sample if the client requires confidentiality.
• Identical prompts before and after, with repeated runs and rates, not screenshots.
• All results, including prompts that did not improve.
• Controls or a comparison group, so platform-wide shifts can be separated from the work.
• Clear settings: assistants, modes, locations, accounts and dates.
• Uncertainty, such as ranges or the number of runs behind each figure.
• Business outcomes where possible, such as AI referral traffic or leads, alongside visibility.

How do you avoid the same trap in your own reporting?

The same selection effect catches in-house teams. A monthly report that highlights "our biggest wins" is picking the prompts that rose most, and some of those rises are chance. Report the whole fixed set first, then the highlights. Show prompts that fell as well as those that rose. When a prompt jumps sharply, rerun it before celebrating, since a large one-round change on a single prompt often reverses the next week. And keep the prompt list fixed for the whole reporting period, adding new prompts as a separate group.

What might an illustrative case look like?

Consider an illustrative agency case study claiming a client went "from invisible to recommended in 90% of ChatGPT answers". Asked for details, the agency shared its ten prompts. Each included the client's niche term, a phrase the client had coined and used across its site. The prompts were chosen at the end of the project, from a list of 60 tested. The buyer reran the ten prompts in a clean account and got 70%. Then they wrote fifteen prompts of their own in the way customers actually describe the problem. The client appeared in 13% of answers, about the same as two competitors. The work had real value for the niche term, but not the broad effect the headline suggested.

What do people ask about AI search case studies?

Are all case studies with selected prompts misleading?

No. Some focus on a specific set for good reason. The problem is selection after the fact, and presenting a narrow set as general visibility.

How many prompts make a credible case study?

Enough to cover the real ways buyers ask, often 30 to 100, fixed in advance and each run several times.

Can I trust screenshots?

Only as examples. A screenshot shows one run of one prompt, and AI answers vary too much for one run to prove anything.