How Do I Measure the Noise Floor for ChatGPT Citation Tracking?

Why does citation tracking need a noise floor?

ChatGPT's answers vary from run to run, even for the same question. SparkToro's study of AI recommendations had hundreds of volunteers run the same prompts nearly 3,000 times. It found less than a 1 in 100 chance that ChatGPT or Google's AI would return the same list of brands twice, and about 1 in 1,000 that the order would match. Citations vary in the same way: which sources are cited, how many and in what order.

So a single run tells you little, and even aggregated metrics wobble. Without knowing how much they wobble by chance, you cannot tell whether a rise from 22% to 27% after a content update is progress or noise. A report that says "our citations went up" without a noise floor cannot answer that question.

How do you run an A/A test?

1. Freeze the setup. Fix the prompt set, the account and its settings, location, language, the model if you can choose it, and whether search is forced. Record everything.
2. Run the full set twice, within a short window such as the same day or two consecutive days, so that nothing real has time to change.
3. Compute every metric for each wave: mention rate, recommendation rate, the share of answers citing your domain, share of voice against competitors, and average rank where you track it.
4. Take the difference for each metric. That is one observation of pure chance.
5. Repeat over several weeks to collect at least five A/A pairs, or use the split-half shortcut below.
6. Set the floor for each metric at about twice the standard deviation of your A/A differences (1.96 × SD). With only five pairs this is a rough estimate, so check it against the split-half shortcut below. Record it with the date and setup.

Measure a separate floor for each assistant and each mode, since search and non-search answers vary differently. Why search and non-search answers should be reported separately is covered in whether ChatGPT visibility reports should separate search, non-search and shopping answers.

What is the split-half shortcut?

If you already run each prompt several times, say ten, you can estimate the floor from one wave. Randomly split each prompt's runs into two halves of five and compute the metric for each half. Repeat the random split a few hundred times with a short script. The spread of the half-versus-half differences estimates the noise floor for a wave of five runs per prompt. It will not capture variation between days, such as model updates, so confirm it with a real A/A test when you can.

How big is a typical noise floor?

It depends on your prompts and on how consistent ChatGPT is for them, so measure your own. As an illustration, we simulated prompt sets where the brand's true appearance rate averaged about 29%. Rates varied widely by prompt: some near zero, some near certain, as is typical. Each wave was compared with an identical rerun:

Two lessons carry over to real tracking. First, one run per prompt gives a floor so high that most real changes vanish in it. Second, the noise shrinks slowly: quadrupling the sample roughly halves it.

In the same simulation, changing the prompt set between waves widened the 50 × 5 floor from ±6 to ±12 points. Keep the prompt set fixed. If you add prompts, report them as a separate group until they have their own history.

How do I know if a change in AI search recommendations is statistically significant or just random variation?

Four checks, in order:

1. Compare like with like. Same prompts, same settings, same mode. If ChatGPT searched more often this month, or a new model rolled out, the comparison is already confounded.
2. Compare prompt by prompt. For each prompt, take its rate in the old wave and the new wave, and look at the differences. Test whether the average difference is above zero with a paired test. A sign-flip permutation test is simple and makes few assumptions: randomly flip the signs of the per-prompt differences thousands of times and see how often the average is as large as the one you observed. Treat the prompt as the unit, not the individual answer, because answers to the same prompt are related.
3. Check it against the noise floor. A change that is statistically significant but smaller than your measured A/A floor deserves suspicion, since the floor includes day-to-day variation the test does not.
4. Replicate. Run the next wave. A change that holds across two consecutive waves is far more convincing than one that appears once.

Also plan for sample size before you make a change. In our simulation, with 50 prompts and five runs each, a paired test would detect a true five-point rise only about a quarter of the time. It would detect a ten-point rise about three times in four. With 100 prompts and five runs, a five-point rise was detected about half the time and an eight-point rise nearly nine times in ten. If the change you hope for is small, you need a bigger sample, or you will conclude "no effect" when there was one.

Watch for multiple comparisons as well. If you track 20 competitors across five assistants, that is 100 comparisons. At the usual 5% threshold, about five will look significant by chance alone. Decide in advance which metrics count, or use a stricter threshold.

How do you lower the noise floor?

• Run more prompts and more runs per prompt, accepting that halving the noise costs about four times the sample.
• Keep the prompt set and settings fixed, and separate modes and assistants.
• Report rolling averages, such as the last four weekly waves, for trend lines, while keeping single-wave results for audits.
• Prefer steadier metrics for decisions. Whether your brand is mentioned varies less than which of your URLs is cited or your exact rank. Use the steadier metric to judge progress, and the noisier ones to diagnose.
• Re-measure the floor after model changes, when ChatGPT's answer format changes, and at least quarterly.

Ways to reduce variation in how tests are run, such as fresh sessions and fixed accounts, are covered in whether to use temporary ChatGPT chats for an unbiased brand visibility test.

What might an illustrative case look like?

Consider an illustrative software company that updated its comparison pages and saw the share of ChatGPT answers citing its domain rise from 22% to 27% the next week. The team had 50 prompts with five runs each. Its A/A tests over the previous month put the noise floor at about six points, and a paired permutation test on the new wave was not significant. Rather than announcing a win, the team ran two more weekly waves: 28% and 29%. Over three waves, the rise held, and the paired test on the combined waves was significant. The team reported the gain with its size and uncertainty, and added 50 prompts to the tracking set to lower the floor for the next test.

What do people ask about noise in AI citation tracking?

How many runs per prompt do I need?

At least three, preferably five or more for prompts that matter. One run per prompt produces a noise floor too high to detect most real changes.

Does using the API reduce noise?

The API gives more control over settings, but its answers can differ from the ChatGPT app, as covered in why ChatGPT API answers differ from the web app. Measure a separate floor for each.

Can a tool's dashboard tell me whether a change is real?

Only if it shows the uncertainty around each number. Ask how many prompts and runs sit behind each metric, and whether the vendor measures run-to-run variation.