Why Do LLM Results Change When You Repeat the Same Prompt?

How much do AI answers vary between repeated runs?

A lot, and in a particular pattern:

• Which brands appear varies moderately. Strong candidates show up in most runs; weaker ones come and go.
• The order of brands varies a great deal. Being named first in one run says little about the next.
• Which pages are cited varies at least as much as order, because it depends on what the search step retrieves each time.
• Wording varies almost every time, even when the substance is similar.

So a brand can be very consistent in one sense, named in nearly every answer, while its position and cited pages keep changing.

What did SparkToro's research find about AI answer inconsistency?

One of the most-cited public tests of this came from SparkToro, published in January 2026 with runs in November and December 2025. Its study had 600 volunteers run 12 recommendation prompts through ChatGPT, Claude and Google's AI results nearly 3,000 times. Key findings:

• In ChatGPT and Google's AI, the chance of two answers containing an identical list of brands was under 1%, and the chance of the same list in the same order was about 0.1%.
• Some brands were still highly consistent. In one hospital prompt, a leading cancer centre appeared in 69 of 71 ChatGPT answers, but was named first in only 25 of them.
• The prompts people wrote for the same need were very different from each other, yet the answers drew on a similar set of leading brands: across 994 headphone answers, Bose, Sony, Sennheiser and Apple appeared 55–77% of the time. The tools captured the shared intent; what changes the list is a different need or constraint, not different wording.
• The authors suggested that seeing an engine's full set of likely recommendations for one prompt could take 60–100 runs.

The practical conclusion was not that tracking is pointless, but that visibility should be reported as how often a brand appears across many runs, never as a rank.

Why do temperature, sampling and server routing make answers vary?

Several layers contribute:

• Sampling. A language model produces text one token at a time, choosing each from a probability distribution. Unless it always takes the single most likely token, different runs follow different paths, and early differences compound.
• Temperature and similar settings. Temperature controls how spread out those choices are. Consumer apps set these for you, and they are generally tuned for varied, natural answers rather than identical ones.
• Inference is not perfectly deterministic. Even with randomness turned down, the numerical results of running a model can differ slightly depending on how requests are batched together on the server. Research published in 2025 by Thinking Machines Lab traced much of this to inference systems that are not "batch invariant". Small numerical differences can change which word wins, and the answer diverges from there.
• Routing. Apps may send requests to different model versions, sizes or configurations depending on the question, load and plan, and may update models without notice.
• Retrieval. When the engine searches the web, the search query it writes, the results it gets and the passages it selects can all differ between runs.

For brand visibility, retrieval often matters most, because it decides which sources the answer is built from.

Why do LLM citations change between runs, and can citation tracking be reliable?

Citations depend on the retrieval step, which varies more than the model's underlying knowledge of a category:

• the engine may rewrite your question into different searches on each run;
• search results shift with time, location and index updates;
• several pages may answer the question about equally well, and the engine picks different ones;
• the number of sources shown per answer varies.

Citation tracking can still be reliable if it measures the right thing. The question is not "which page was cited?" but "in what share of runs was our domain, or this page, cited?". With enough runs, that share is stable enough to track. Domains also tend to recur more consistently than specific URLs, so report both.

How consistent are ChatGPT citations for the same prompt?

There is no universal figure, because consistency depends on the category, the question and how many good sources exist. Typical patterns when you run the same question repeatedly:

• a few domains appear in most runs; these are the engine's preferred sources for that question;
• a long tail of domains appears once or twice;
• your own site may be cited through different pages on different runs.

Measure it for your own questions: run each one ten times, list the cited domains and URLs, and count how often each appears. The domains that appear in most runs are the ones worth targeting; see how to find high-frequency AI-cited sources.

Why does ChatGPT cite a different page about your business every time?

Usually because several of your pages are similarly relevant to the question: a homepage, a service page, a blog post and a location page may all mention the same service. The engine picks whichever passage fits best on that run.

This is not necessarily a problem, but it spreads your citations thin and can mean an outdated page gets cited. To make it more consistent:

• create one clear, complete page per main question or service, and link other pages to it;
• update or redirect older pages that say similar things;
• state the key answer early and plainly on the page you want cited.

Why does brand visibility vary so much between models?

Different models, even from the same company, have different training data, knowledge cutoffs and tendencies to search the web. One model may answer from memory and name established brands; another may search and cite recent sources that name newer ones. Apps also switch between models as they release updates or route by question type. This is why tracking should record which model and mode produced each answer, and why a sudden shift in results may reflect a model change rather than anything you did.

Why are AI recommendations more unpredictable than Google rankings?

A Google results page is produced by a ranking system that, for a given query, place and moment, returns a fairly stable ordered list. It changes as the index and signals change, but not every time you search.

An AI answer adds a generation step on top of retrieval. It is written fresh each time, can draw on different sources each time, and names a handful of businesses in prose rather than listing ranked results. More moving parts produce more variation. For classic search, rank tracking works; for AI answers, you need rate tracking. How the two measurement approaches compare is explained in how to measure success in GEO compared with SEO.

What is run-to-run variance, and how should you measure it?

Run-to-run variance is how much results change when nothing but the run changes: same question, engine, mode, location and day. Measure it with a few simple metrics:

Always show the uncertainty. With a 95% Wilson interval, being named in 4 of 5 runs is consistent with a true rate anywhere from about 38% to 96%. At 16 of 20 runs, the range narrows to about 58% to 92%, and at 40 of 50 runs to about 67% to 89%. That is why five runs can separate "almost always" from "rarely", but cannot detect a change from 60% to 70%.

Practical rules:

• use at least five runs per question per engine for routine tracking, more for questions you report individually;
• pool related questions to get larger samples for trends;
• compare periods using the same number of runs and the same settings;
• treat changes smaller than the interval as noise.

How many runs to use in different situations is covered in how many times to test a prompt to verify visibility.

What does run-to-run variance look like in a tracking sheet?

Here is an illustrative log of ten runs of one question, "best project management tool for a small design agency", in one engine on one day. Each row lists the brands named, in order:

Brand A appears in all 10 runs but is first in only 5. Brand B appears in 8, first in 3. C appears in 5, D in 4 and E in 2. No two runs are identical, yet the picture is clear: A is the engine's default recommendation, B a strong second, and D and E are occasional. A single screenshot of run 10 would have suggested D leads, which is wrong.

How do you tell a real change from normal variance?

Compare periods with the uncertainty in view. Suppose you were named in 12 of 20 runs last month and 16 of 20 this month. The intervals, roughly 39–78% and 58–92%, overlap heavily, so the rise may be noise. Before calling it a change, look for:

• the same direction across several questions in the same group, not just one;
• the same direction in more than one engine;
• a change that persists into the following period;
• no matching shift in your control questions, which would suggest an engine update affecting everyone.

Pooling related questions helps. Twenty questions with five runs each gives 100 answers per period; even then, a move from 30% to 45% sits close to the margin of error (about ±13 points if every answer were independent, and repeated runs of one question are not). Treat it as a possible change until the next period confirms it, and add questions rather than runs when you need more precision.

Can you make your own brand appear more consistently?

You cannot change how an engine samples, but you can change how strong the evidence for your brand is, and that affects consistency. In the SparkToro data, a few brands appeared in almost every run; these are typically category leaders with a great deal written about them. Brands with thin or mixed evidence appear only some of the time.

The levers are the usual ones: clear pages that answer the question, consistent facts across your site and listings, and presence on the third-party pages engines cite for the question. As evidence builds, your appearance rate tends to rise and become steadier, even though order and cited pages keep moving.

What do people ask about repeated AI answers?

Can I make ChatGPT give the same answer every time?

Not in the consumer app. Through the API you can lower randomness, but even then small variations remain, and search results still change.

Does variation mean AI visibility tracking is pointless?

No. It means tracking must use repeated runs and report rates with ranges. Brands that are consistently strong or weak show clearly.

Should I worry if my brand disappears from one answer?

Not on its own. Check it over several runs. A drop across many runs and questions is worth investigating.

Do all AI engines vary equally?

No. Engines differ in how often they search, how many sources they cite and how they are configured, so measure variance per engine.

Is run-to-run variation worse for smaller brands?

Usually, yes. Brands with limited evidence sit near the edge of the engine's shortlist, so they drop in and out between runs. Category leaders appear in almost every run. For smaller brands this means more runs are needed to measure visibility accurately, and steady evidence-building is what moves them from occasional to regular mentions.