How to Manually Track AI Visibility by Testing Prompts

Why can't one check tell you whether AI recommends you?

Because AI answers to recommendation questions change from one run to the next, often a lot.

SparkToro had 600 volunteers run 12 recommendation prompts through ChatGPT, Claude and Google's AI results 2,961 times in late 2025. For ChatGPT and Google's AI there was less than a 1-in-100 chance that two answers listed the same set of brands, and roughly a 1-in-1,000 chance that they listed them in the same order; Claude was only slightly more consistent. In one prompt about West Coast cancer hospitals, City of Hope appeared in 69 of ChatGPT's 71 answers but was named first in only 25.

Two lessons follow. A single screenshot of ChatGPT naming you, or ignoring you, is one draw from a wide range of possible answers. And being mentioned and being mentioned first are different results, so track them separately.

How do you build a fixed set of buyer questions?

Start from the questions a buyer asks before they know your name, written the way they would type them. Fifteen to thirty questions is enough for one business or product line; that is the baseline you will rerun every month, so keep it small enough to finish.

Build the list from five kinds of question:

Source the wording from sales-call notes, support tickets, review text, onboarding forms and the "People also ask" box, not from your keyword list. Real phrasing varies more than you would expect: in the SparkToro research, 142 prompts that people wrote for one need were so different that their average semantic similarity was 0.081. For a deeper walkthrough, see how to choose the right prompts to track.

Once written, freeze the list. Give each question an ID (Q01–Q25) and never edit the wording between rounds; add new questions as a separate set instead.

How do you avoid vanity prompts?

A vanity prompt contains your name or leads the engine toward you: "Is Acme Payroll good?", "What does Acme do?", "Why is Acme the best option for startups?". They show how an engine describes you once someone already knows you, which matters for accuracy, but they tell you nothing about whether new buyers will ever hear of you. Keep branded prompts in a separate, smaller list and report them separately; mixing them in makes every visibility number look better than it is. See whether brand-name prompts inflate AI visibility scores.

A simple test for each question: would a buyer who has never heard of you type this? If not, it belongs on the branded list.

Which questions matter most for B2B buyers?

For B2B, weight the list toward late-stage questions, because that is where a shortlist is formed: requirement questions ("SOC 2 compliance software for a 50-person fintech"), comparisons against the two or three vendors you lose deals to, "alternatives to [market leader]", and questions about price, implementation time and integrations. Include one or two questions for each buying-committee role you sell to; a CFO and an IT lead ask about the same product in different words and often get different vendors back.

Which AI engines should you include?

Include the engines your buyers use, and record for each answer whether it came with sources. That detail changes what you can fix.

Google's documentation on AI features and your website says AI Overviews and AI Mode may issue several related searches for one question, may use different models so the answers and links vary, and show Overviews only when Google judges them useful, so they often don't appear. Log AI Overviews and AI Mode as separate engines rather than one "Google" result.

How do you keep each test clean?

A manual test is only comparable over time if the conditions stay the same:

1. Start a new conversation for every question. Earlier turns change the next answer.
2. Remove personal context. Turn off saved memory and custom instructions, or use a temporary or logged-out session; see whether to use temporary chats for an unbiased test.
3. Paste the exact wording from your list; never retype from memory.
4. Hold location and language constant. If location matters, put the city in the question and note where you ran it; see how to test AI visibility for different cities.
5. Log the first answer before clicking any follow-up suggestion.
6. Write down the engine and mode (for example a reasoning mode versus the default), since they can differ.

If you later automate with an API, expect some difference from the apps; see why API answers differ from the web app.

How many times should you run each question?

Enough that one lucky or unlucky answer cannot flip your conclusion. SparkToro suggests 60–100 runs of a question to see an engine's full set of recommendations, which suits research, not a monthly check. For routine tracking, three to five runs per question per engine is a workable compromise, reported as a count: "named in 4 of 5 runs".

Do the arithmetic before you start:

More on the trade-off in how many times to test a prompt, and on why answers move at all in why LLM results change when you repeat the same prompt.

What should you record for every answer?

One row per answer. These columns are enough to answer every question you will be asked later:

The last four columns are the ones people skip, and they are the ones that tell you what to fix. A wrong description, such as "residential" when you only do commercial work, usually points to an outdated page or listing that the engine trusted.

How do you set up the tracking spreadsheet?

Use one workbook with three tabs. It works the same in Google Sheets and Excel.

• Questions: ID, exact wording, type (category, requirement, comparison, problem, local), and whether it is branded.
• Runs: one row per answer with the columns above. Put Question ID in column C, Engine in column B, "You named?" in column F, Businesses named in column H, Competitors named in column I and "Your page cited?" in column K.
• Summary: questions down the side, engines across the top.

The core formulas, assuming the Summary tab has question IDs in column A and engine names in row 1:

• Appearance rate for one question on one engine: =COUNTIFS(Runs!$C:$C,$A2,Runs!$B:$B,B$1,Runs!$F:$F,"Y")/COUNTIFS(Runs!$C:$C,$A2,Runs!$B:$B,B$1)
• Question coverage (share of questions where you appeared at least once): =COUNTIF(B2:B26,">0")/COUNTA(B2:B26)
• Share of voice across all answers: =COUNTIF(Runs!F:F,"Y")/SUM(Runs!H:H)
• Citation share: =COUNTIF(Runs!K:K,"Y")/COUNTIF(Runs!J:J,"<>")

Keep the raw Runs tab untouched after each round and copy it into a dated tab, so any number in a report can be traced back to the answers behind it.

How do you compare your visibility with competitors?

Pick the three to five competitors that appear most often in the "Competitors named" column, not the ones your sales team fears most; the two lists often differ. For each, count how many answers named them: =COUNTIF(Runs!I:I,"*Competitor A*")/COUNTA(Runs!B:B), and put their rate next to yours for every engine.

The useful view is per question, not overall. A table of questions against competitors shows where one rival owns a whole question type, for example every comparison question, and those rows are where to start. Then open the pages cited in those answers: they show what the engine trusted instead of you. For more, see how to measure AI share of voice.

How do you track which pages AI cites each week?

Citations are easier to act on than mentions, because they name the exact page an engine relied on. Perplexity shows sources for nearly every answer, so it is the best engine for a weekly citation check. ChatGPT, Claude and AI Overviews show links only when they search or when an Overview appears, so log "no sources" as its own value rather than leaving the cell empty.

A weekly routine that takes under an hour:

1. Run your ten most important questions in Perplexity, once each.
2. Paste every cited URL into the Runs tab.
3. Tag each URL as your page, a competitor's page, a directory or marketplace, a review site, or media.
4. Count how many answers cited any of your pages, and which of your pages they were.

Over a few weeks this shows which of your pages engines actually use, whether new pages start getting cited, and which third-party sites keep winning the questions you care about. Your homepage appearing instead of the page that answers the question is a common and fixable finding.

What can you learn from a 30-minute check?

A 30-minute check tells you whether you have a problem, not how big it is. Take your five most commercial questions, run each once in the two engines your buyers use most, and note for each answer whether you were named, who was named, and whether anything said about you was wrong. That is ten answers at about three minutes each.

If you are named in none of the ten, you almost certainly have a visibility gap worth a full baseline. If the description of you is wrong anywhere, fix that first, because it affects every question. Treat everything else as a hint, since one run per question is not enough to measure.

How do you run a five-question mini audit?

A mini audit is the smallest version that produces numbers you can trust a little. Use five questions (two category, one requirement, one comparison, one problem), three engines and three runs each: 45 answers, about 60–90 minutes. Report each question as "named in X of 9 answers" and list every competitor named at least three times. It is enough to decide where to look first, and it becomes the seed of your full question set.

What does a one-week test look like?

For a full baseline of 20 questions, five engines and three runs (300 answers), spread the work so most days take one to two hours; Day 4, with two engines, takes two to four:

Run the whole test within the same week so a model update does not land in the middle of your baseline.

Should you track AI visibility alongside SEO?

Yes. Put your normal search ranking for the closest keyword next to each question in the Questions tab. Engines that search the web retrieve pages from search indexes, so the two are related, but not identical: you will find questions where you rank well and are never mentioned, and the reverse. Those mismatches are the most useful rows in the whole sheet, because they separate a content or authority problem from a retrieval problem. Take the baseline in the same month as a normal SEO report so the two can be read together.

Example: a home security company across five engines

This is an illustrative example, not real data. Suppose a regional monitored-alarm company tests 20 questions such as "best home security system for renters in Austin", "monitored alarm company with no long-term contract", "ADT alternatives" and "is professional monitoring worth it", across ChatGPT, Perplexity, Gemini, Google AI Overviews and Claude, three runs each.

A typical pattern in a result like this:

• The company is named in some local "best" questions but in none of the comparison questions, where national brands dominate.
• Perplexity cites a review site's "best alarm companies in Austin" list in most answers, and the company is not on it.
• One engine describes the company as installation-only, which traces back to an old directory listing.
• AI Overviews appear for only some questions, so the "no Overview" rows matter as much as the named rows.

The fixes follow directly from the log: correct the directory listing, get onto the review site's list, and publish a comparison page that answers the "alternatives" question in the company's own words.

How often should you repeat the test?

Monthly is enough for most businesses. Rerun sooner after a change you want to measure: a new service page, a rewritten pricing page, a new listing, a site migration. Keep the question set frozen between rounds; if the questions change and the numbers move, you cannot tell which caused it. See how often to track AI visibility. Note outside events next to each round, such as a major model release, because visibility can move without any change on your side.

When does manual tracking stop being practical?

Manual tracking works well for a first baseline: one business, one location, 15–30 questions, three to five engines. It gets hard to sustain when the number of engines, runs, locations or clients grows, because the answer count multiplies, and when you need to show before-and-after evidence for a fix. At that point the options are a script that calls the engines' APIs (cheaper, but API answers can differ from the apps), a spreadsheet with an automation tool, or a dedicated tracker. Keep the same rules whichever you choose: fixed questions, repeated runs, counts instead of single answers, and cited pages next to every result.

What do people ask about tracking AI visibility by hand?

Should I put my business name in the test questions?

Not in your main set. Questions that name you measure how AI describes you once someone knows you. The questions that bring new customers are the ones where the buyer has never heard of you, so keep branded prompts in a separate list and report them separately.

Is a screenshot enough evidence?

A screenshot proves what one answer said. Store the date, exact question, engine, mode and cited links with it; without them you cannot rerun the same test or show what changed after a fix.

What if an engine recommends nobody for a question?

Log it as its own result. A question where no business is named, or where only directories and marketplaces are cited, is often the easiest place to become the answer, because there is no incumbent to displace.

Why did my result change when I changed nothing?

AI answers vary from run to run, models are updated, and the pages an engine retrieves change as other sites publish. That is why one round is a baseline rather than a verdict, and why the same questions need to be rerun on a schedule.