Is Asking LLMs Directly a Valid Way to Measure Visibility?

What counts as asking LLMs directly?

Two very different things get described the same way:

Observation treats the model as the thing being measured. Self-report asks the thing being measured to do the measuring, which it cannot do.

Why are self-report and leading questions not valid?

• No access to usage. A model does not know how often it has named your brand to other users. It has no log of past answers to consult.
• Invented statistics. Asked for numbers about its own behaviour, a model may produce plausible figures with no basis.
• Agreement with the framing. Models tend to go along with how a question is framed, a tendency researchers call sycophancy. "Would you recommend [brand]?" invites a yes that says little about whether the brand is recommended unprompted.
• After-the-fact explanations. When asked why it chose one business, a model generates a reasonable-sounding explanation, which may not reflect how the answer was actually produced. Cited sources, when present, are better evidence.

Branded questions such as "What is [brand]?" are useful for a different purpose: checking whether facts about you are correct. They are accuracy checks, not visibility measures.

When is observing answers a valid measurement?

Direct observation is sound when these conditions hold:

1. Representative questions. They reflect real buyer needs and wording, from sales, support and search evidence, and most do not include your name.
2. Repeated runs. Each question is asked several times per engine, since answers vary between runs, and results are reported as counts.
3. Clean, recorded conditions. Fresh sessions without personal history, a stated location and language, and the engine, model and mode noted.
4. Consistent method over time. The same questions and settings each period, with any changes versioned.
5. Evidence kept. Full answers and cited sources are saved, so every number can be checked.
6. Consistent classification. Clear rules for what counts as recommended, mentioned or negative.

Under these conditions, direct checks measure something real: how often the engine recommends you for the questions you chose. How many runs are enough is covered in why LLM results change when you repeat the same prompt.

What can direct checks not tell you?

Even done well, observation has limits:

• What real users ask. Real buyers phrase needs in countless ways, and your questions are a sample you chose. ChatGPT, Claude and Perplexity do not show you the prompts people type; Google counts AI Mode questions in Search Console's Performance report but mixes them with ordinary searches, and Bing's AI Performance report shows only the grounding queries its AI generated.
• How often questions are asked. Direct checks show what happens when a question is asked, not how many people ask it.
• Personalised answers. Real users have histories, locations and settings you cannot fully reproduce.
• Outcomes. Whether a recommendation leads to a visit, a call or a sale needs analytics, attribution and sales data.

So direct observation measures exposure on a defined question set. It should be paired with outcome measures before drawing business conclusions.

How do you make manual checks more valid?

Small habits make a big difference:

• write the question set before looking at any answers, so you do not pick flattering questions;
• include a few control questions whose answers should not change, to spot engine-wide shifts; see what a control prompt is;
• use a separate test account or temporary chats, never your everyday account;
• have a second person classify a sample of answers and compare;
• log failed or unusual runs rather than repeating until you get a "normal" answer;
• record the date and model, because engines change without notice.

How does direct observation compare with other ways of measuring?

No single method is enough. Direct observation, manual or automated, is the only one that shows what the engines actually say, which is why it sits at the centre of most measurement plans. How branded search fits in is covered in whether branded search volume can measure LLM visibility.

What do people ask about measuring visibility by asking LLMs?

Is typing "best [thing]" into ChatGPT a good visibility check?

It is a start, but one short question, asked once, in your own account, is weak evidence. Use buyer wording with conditions, several runs and clean sessions.

Can I ask ChatGPT what sources it uses for my industry?

You can ask, but the answer describes typical sources in general terms and may be invented. Logging the sources it actually cites across many answers is more reliable.

Is using the API instead of the app still valid?

Yes, as a consistent measurement, but API answers can differ from what app users see. Check a sample in the apps.

How many questions make a valid check?

For one market, 20–40 questions with several runs each is a practical minimum for tracking. Fewer questions can still be useful for a quick diagnosis.