How to Retest the Same AI Prompts After Updating Your Website

Why is retesting harder than it looks?

• Answers vary by run. The same prompt can name different brands and cite different pages each time.
• Platforms change. Models, search partners and citation behaviour shift, sometimes sharply, without notice.
• Settings change answers. Logged-in accounts, saved memory, location, language and model choice all affect results.
• Pickup takes time. An update cannot appear in answers until crawlers fetch the new version, and cited sources churn: Scrunch found half of the average source's citations disappeared within about four and a half weeks.

What should you do before changing anything?

1. Freeze the prompt set. Choose the prompts the update should affect, written the way buyers ask them. Keep the exact wording. Choosing prompts is covered in how to choose the right prompts to track for AI visibility.
2. Add control prompts on related topics the update should not affect.
3. Record the settings you will use every time.
4. Run a baseline: several runs per prompt per engine, spread over a week or two.
5. Save the raw answers and cited URLs, not just a yes or no.
6. Start a change log with each page changed, what changed and the publication date.

Which settings should stay constant?

Why API answers differ from the app is covered in why ChatGPT API answers differ from the web app when tracking citations.

How many runs, and how often?

More runs give more reliable results. For important prompts, 10 runs per engine per check is a sensible minimum. Because citations churn within weeks, a result from week two can differ from week eight. The trend across checks is what counts.

How to track AI citation changes after updating specific pages

1. Map pages to prompts. For each updated page, list the prompts it should answer and the fact or passage it should supply.
2. Confirm recrawling. Check server logs for visits from Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot after the change date. Request indexing in Search Console, and submit the URL through IndexNow for Bing and other participating engines.
3. Use a changed fact as a marker. If the update changed a price, date or specification, check whether answers quote the new value or the old one. That is direct evidence of pickup.
4. Record page-level citations for each run: whether the page is cited, for which prompts and alongside which competitors.
5. Read first-party reports. Bing Webmaster Tools' AI Performance report shows citations per page for Copilot, and Search Console's generative AI performance report shows AI Overview and AI Mode impressions per page.
6. Compare with control pages that you did not change during the same period.

How can you tell whether the update caused the change?

• Compare against controls. If citations rose for updated pages and control pages alike, the platform changed, not your page.
• Check the size of the change. Moving from 2 of 10 runs to 3 of 10 is within normal variation. Moving from 1 of 10 to 7 of 10, repeated across checks, is a real change.
• Check the new content is what's quoted. If answers now repeat the new fact or passage, the update is the likely cause.
• Watch for platform-wide events. Industry trackers often report sudden citation drops or model changes. Note them in your change log.
• Avoid changing several things at once on the pages you want to learn from, or you will not know which change worked.

Whether a specific error has been fixed is covered in how to tell whether an AI error was corrected or is still appearing.

What should each record include?

For every run, record:

• date and time;
• engine, model and mode;
• account state, location and language;
• the exact prompt;
• the full answer text;
• brands named, and in what terms;
• cited URLs;
• whether the answer includes your changed fact, and which version.

A spreadsheet works for small programmes. Larger ones benefit from a tool that stores raw answers and exports them. Whichever you use, keep the records for at least a year, since seasonal changes and model updates only become visible when you can compare the same prompts across several quarters.

What might an illustrative retest look like?

Consider an illustrative accounting software company that rewrites its pricing page to show plan prices in text and adds a comparison with its main competitor. Before publishing, it runs 12 prompts, such as "how much does [product] cost" and "[product] vs [competitor]", ten times each in four engines. It also runs six control prompts about unrelated features. At baseline, answers quote an old price in 40% of runs and cite the pricing page in 10%. Logs show the page recrawled by Bingbot and OAI-SearchBot within four days. Six weeks later, the new price appears in 70% of runs and the pricing page is cited in 45%. Citation rates for the control prompts are unchanged. The company concludes that the rewrite worked and applies the same approach to its feature pages.

What do people ask about retesting AI prompts?

How long should I wait before retesting?

Until the updated pages have been recrawled, usually days to two weeks, then check again at four to six weeks.

Can I retest with a tool instead of by hand?

Yes, if the tool keeps prompts and settings fixed, runs each prompt several times and stores raw answers and citations.

Why did my visibility drop after an update?

Check the control prompts first. If they dropped too, the platform changed. If only updated pages dropped, review what the update removed.

Should I change the prompts over time?

Keep the core set fixed for comparisons. Add new prompts separately, so trends stay comparable.