What Is a Golden Prompt Set for AI Visibility, and Why Keep It Fixed?
A golden prompt set is a fixed, versioned list of the buyer questions that matter most to your business. It is run on a regular schedule with the same assistants, accounts, locations and settings every time. Keeping it fixed means month-to-month changes in your results reflect changes in the assistants and in your own work, not a different list of questions. Fixed sets have weaknesses: over time they can drift from how buyers actually ask, and teams can end up optimising for the test. Random or rotating prompts represent the wider market better but are noisier. The practical answer is a fixed core for trend reporting plus a rotating sample for discovery, with clear rules for when and how the core changes.
Why keep it fixed?
- Comparability. If this month's questions differ from last month's, a rise or fall may simply reflect easier or harder questions.
- Lower noise. AI answers vary from run to run, and changing the questions adds a second source of variation. In a simulation we ran, swapping prompt sets between rounds roughly doubled the chance swing in appearance rates for a 50-prompt set, from about 6 to about 12 points.
- Honesty. A fixed list chosen in advance cannot be quietly adjusted to flatter results. How selection inflates results is covered in how to detect cherry-picked prompts in an AI search case study.
- Attribution. With a fixed set and control prompts, you can separate your own effect from platform-wide shifts, as covered in what a control prompt is in AI search visibility measurement.
What goes into a golden set?
- Real buyer questions, drawn from sales calls, support tickets, interviews and search data. How to collect them is covered in how to turn customer interviews, sales calls and support tickets into AI search prompts.
- Coverage across the buying journey: problem questions, category questions, comparisons, pricing and objections.
- Mostly unbranded prompts, with branded ones kept as a separate group.
- Control prompts that your planned work should not affect.
- Enough prompts and runs to detect the changes you care about, often 50 to 150 prompts, each run several times.
Record each prompt's source, tags, date added and version.
Frozen prompt set vs random prompts: which is better for measuring AI search visibility?
| Frozen set | Random prompts from a large pool | |
|---|---|---|
| Best for | Measuring change over time | Estimating overall visibility across the market |
| Noise | Lower, since the questions stay the same | Higher, since each sample differs |
| Representativeness | Can drift as buyer language and products change | Stays closer to the wider market if the pool is maintained |
| Risk | Optimising for the test: content tuned to exact wording | Harder to explain month-to-month changes |
| Discovery | Finds nothing new | Surfaces new questions and competitors |
Neither is better for every purpose. A frozen set answers "are we improving?". A random sample answers "how visible are we overall?". Most teams need both answers.
Fixed AI prompt panels vs rotating prompt panels for month-over-month reporting
For month-over-month reporting, use a fixed panel, or a rotating design where most of the panel overlaps between months. Survey researchers have used rotating panels for decades. The US Current Population Survey, for example, keeps households in its sample for four months, rotates them out for eight, then brings them back for four. As a result, about three-quarters of the sample overlaps from one month to the next. The overlap keeps monthly changes precise while the rotation keeps the sample fresh.
A version for AI visibility:
- A fixed core, perhaps 70% to 80% of prompts, reported month over month.
- A rotating share, the remaining 20% to 30%, drawn from a larger pool each month and reported separately as a discovery sample.
- Promotion rules: a rotating prompt that keeps revealing important gaps can join the core at the next version change.
Report the core's trend as the headline. Report the rotating sample as "wider market check", with its larger uncertainty stated.
How do you change the core without breaking the trend?
Buyer questions, products and competitors change, so the core needs occasional updates. Do it deliberately:
- Version the set, such as "core v2", and change it no more than once or twice a year unless something major changes.
- Run old and new versions side by side for two or three rounds before switching. The overlap lets you link the two series.
- Never remove prompts because results are bad. Remove them only if they no longer reflect real buyer questions, and record why.
- Keep retired prompts available for occasional checks.
- Note the version change on dashboards and in reports.
How often should the golden set run?
Match the schedule to how fast you act on results. Weekly runs suit teams making frequent content changes or watching a launch. Monthly runs suit most brands. Whatever the rhythm, keep it steady: same day of the week, similar time of day and the same accounts, since answers can differ with timing and personalisation. Run every prompt several times per round rather than once, and report rolling averages across rounds for trends. Record the date, assistant, mode and any visible model name with every run, so later changes can be explained.
Who should own the golden set?
One named person, usually in the measurement or search team, should own the list, its version history and the rules for changing it. Content, product and sales teams can propose prompts, but changes should go through the owner at a version change. Without an owner, prompts get added and removed ad hoc, and the trend quietly stops meaning anything.
How do you avoid optimising for the test?
If your content team writes pages to match the exact wording of your golden prompts, visibility on the set can rise while real buyer visibility does not. Guard against it:
- Add paraphrase variants of important prompts, and rotate which variant is run.
- Watch the rotating sample. If the core improves but the rotating sample does not, suspect overfitting.
- Keep the full prompt list with the measurement team, and share themes rather than exact wording with content writers.
What might an illustrative case look like?
Consider an illustrative software company that changed its tracked prompts every month, picking "whatever seemed relevant". Its visibility reports swung between 18% and 35% with no clear pattern. It switched to a 100-prompt core drawn from sales calls, fixed for a year, plus 25 rotating prompts each month from a pool of 400. Over the next six months, the core showed a steady rise from 21% to 29%, beyond the run-to-run noise. The rotating sample moved from 17% to 20%, well within its much larger noise, since 25 new prompts a month can swing by 15 points or more by chance. So it could not yet show whether the gains carried over to wider buyer questions. As a precaution, the team added paraphrase variants for the ten most important prompts and kept collecting rotating data until it had enough to compare.
What do people ask about golden prompt sets?
How many prompts should the core contain?
Enough to cover your main buyer questions and detect meaningful changes, commonly 50 to 150, each run several times.
Should the golden set include competitor names?
Include comparison prompts that buyers really ask, but keep them tagged so they can be reported separately.
What if an assistant changes its model?
Keep the set fixed and annotate the change. A fixed set is what makes the model change visible.
Published by The Kovmere Team · updated Sep 29, 2026 · More resources