What Are the Limitations of Checking Only 20 Buyer Questions Across Five AI Search Engines?
Checking 20 buyer questions across five AI engines is good at showing where you are named, who is named instead and which pages the answers rely on. Its limits are real, though: 20 questions cannot cover every way buyers ask, a set that small cannot reliably detect small changes, the result depends heavily on which 20 you choose, and a test run through fixed settings misses personalised, multi-turn and app-specific answers. Treat it as a sharp diagnostic for the questions you chose, not as a census of your AI visibility.
What can a 20-question check do well?
It finds gaps that matter. If competitors are named for a need you serve and you are not, a handful of runs will show it, and the cited pages will usually show why. It compares engines on identical questions, it catches wrong facts being repeated, and it points to the few pages, yours and other people's, that decide the answers in your category. For most businesses that is the most useful information they can get for the effort.
What are its main limits?
| Limit | What it means | How much it matters |
|---|---|---|
| Coverage | 20 questions sample the ways buyers ask; they do not cover the long tail | High for broad categories, low for narrow ones |
| Precision | Small changes between readings can be chance | High when you track month-to-month movement |
| Selection | Results depend on who chose the questions and how | High if the questions were written to flatter |
| Variation | The same question gives different shortlists on different runs | Medium, if each question runs several times |
| Settings | One location, language and buyer type per set | High for multi-market or multilingual businesses |
| Conversation | Single questions miss follow-ups, memory and personalisation | Medium; real chats narrow shortlists over several turns |
| Collection method | API or simulated sessions can differ from what one person sees in an app | Medium; patterns hold better than exact wording |
Why can't 20 questions cover the market?
Buyers phrase the same need in hundreds of ways, and the need behind the words changes the answer more than the wording does. SparkToro asked 142 people to write a question with the same intent and found the prompts almost entirely different from one another, yet the leading brands still appeared in 55–77% of the 994 answers. Different needs, such as gaming versus podcasting headphones, produced very different lists. So a small set works when it spans the distinct needs and constraints in your category; it fails when it repeats one need in 20 phrasings.
Why are small changes hard to read?
With 20 questions, one question moving from "named" to "not named" shifts your rate by five percentage points. AI answers vary from run to run: in SparkToro's tests, the chance of ChatGPT or Google's AI giving the same brand list twice was under 1 in 100, with Claude only slightly more consistent. Several runs per question help, but repeated runs of one question are not independent evidence about the market, so they narrow the noise less than the run count suggests. In practice, treat a change of one or two questions as unconfirmed until it holds over another reading, and look at which questions moved rather than at the total. The arithmetic is set out in how to calculate AI visibility as an appearance rate across repeated runs.
How does question choice bias the result?
Questions that mention your niche, your feature names or your city's smallest suburb make you look visible. Questions chosen by a vendor that wants to sell you fixes can make you look invisible. Both are the same flaw: the set was not built from how buyers ask. Guard against it by writing questions from customer wording before looking at any results, keeping brand-name questions separate from the main figures, and freezing the set so it cannot be adjusted to tell a better story later.
What does a fixed-settings test miss?
- Personalisation and memory. Signed-in users with history and saved preferences can see different suggestions.
- Follow-up questions. A buyer who asks "which of those is cheapest?" gets a shorter list than the first answer.
- Location and language. One set reflects one market. A business serving several countries or languages needs a set for each.
- App features. Shopping cards, maps and inline product carousels appear in consumer apps and may not appear in the same form in API answers.
None of these makes a fixed test useless; they mean its findings describe a representative buyer in one setting, not every buyer.
How can you reduce the limits?
- Span needs, not phrasings. Cover each distinct problem, constraint and audience once before adding variants.
- Run each question several times and record how often you appear, not whether you appeared once.
- Freeze the set between readings, and change it deliberately, a few questions at a time, keeping the history of the rest.
- Read per question. A total hides which needs you are winning and losing.
- Add a second set for each important market, language or product line instead of stretching one set thin.
- Cross-check with platform data. Google's Search Console AI report and Bing's AI Performance report show your own pages' appearances, which a question set cannot.
When are 20 questions enough?
For a business with one main market and a focused offer, 20 well-chosen questions usually cover the decisions that matter: which pages to fix, which sources to approach, which engines treat you differently. They are not enough for estimating market share across a large category, tracking dozens of product lines, or proving a small month-to-month improvement. The question of the right number is discussed in more depth in how many times you should test a prompt to verify visibility.
How does our product, Kovmere, handle these limits?
Kovmere, our product, is built on exactly this design, so its limits are the ones above, and it is fair to say how it treats each. A report uses up to 20 buyer questions for one market agreed at checkout: up to 18 that do not name your brand, and up to 2 brand comparison questions kept apart from the main figures. Fewer than 20 can appear when your site does not carry enough evidence to support a question, and the report says so. Each question runs three times across OpenAI, Perplexity, Google Gemini, Anthropic Claude and Google AI Overviews, up to 300 answers, and the answers and the pages they cite is shown so you can check the reading yourself.
The question set is frozen. On Pro, which re-asks the same questions in a baseline and three weekly re-tests each billing period, you can replace up to 5 questions per round after the first baseline; replaced questions start a fresh history and the rest keep theirs. Answers are collected through the engines' search-enabled APIs and Google's AI Overview results with US settings, not from the consumer apps, so personalised and multi-turn answers are outside what it measures. The figures describe the questions it asked; they do not estimate your share of every conversation, and no reading guarantees future citations or recommendations.
What do people ask about the limits of a 20-question check?
Is a 20-question check worth doing at all?
Yes, if the questions span the needs your buyers have. It reveals the gaps and the pages behind them for a fraction of the effort of a large panel, which is what most businesses need first.
Would 100 questions remove the limits?
They would improve coverage and precision, but not selection bias, fixed settings or collection method. More questions written carelessly add cost, not truth.
Why do two checks a week apart disagree?
Mostly because answers vary from run to run. If only one or two questions changed, assume variation until the next reading confirms it.
Should I test in more than one country?
Only if you sell there. Use a separate set per market, because local sources and competitors change the answers.
Published by The Kovmere Team · updated Oct 3, 2026 · More resources