How to Test Whether GEO Changes Worked: Holdouts, A/B Tests and Experiments

Does before-and-after tracking prove that GEO changes caused visibility gains?

No. Before-and-after tracking shows that visibility changed, not why. AI answers move for reasons unrelated to your work:

• Model and retrieval updates: engines change models and search systems regularly. AI Overviews, for example, moved to Gemini 3 in January 2026, and the overlap between their citations and Google's top results changed sharply.
• Competitor activity: new pages, reviews or coverage for rivals.
• Seasonality and news: topics rise and fall.
• Run-to-run variance: the same question can produce different brands on different runs.
• Changes to the sources engines cite: a comparison article updates its list.

Before-and-after data is still worth keeping as context. To make a causal claim, you need a comparison group that experienced the same outside changes without your intervention.

How can you test whether an SEO tactic actually improves AI search results?

Use the same discipline as any experiment:

1. Hypothesis: "Adding a direct-answer opening paragraph increases how often our service pages are cited in ChatGPT and Perplexity."
2. Unit: what you change and measure, usually pages, or the question groups those pages serve.
3. Metric: citation rate or mention rate across repeated runs of a fixed question set, defined in advance.
4. Control: comparable units left unchanged.
5. Duration: long enough for engines to recrawl and reflect the change, usually four to eight weeks after the change goes live.
6. Analysis plan: decide before you start how you will compare the groups and what counts as a meaningful result.

Writing the plan down before you see the results prevents the most common error: finding a story in whatever moved.

What can you randomise in a GEO experiment?

You should not show AI engines one version of a page and people another: showing crawlers content that people cannot see, to influence rankings or answers, is cloaking. And engines do not split traffic for you. So GEO experiments randomise across units instead:

Randomise which units get the change, rather than choosing the most promising ones. Hand-picking produces groups that differ before the test begins.

How do you A/B test GEO content in real campaigns?

In practice, "A/B testing" GEO content means split-page testing:

1. Pick a template with many similar pages, such as location pages, product pages in one category, or help articles.
2. Write the questions each page should answer, two or three per page, in buyers' words.
3. Measure a baseline for all pages: run every question three times in each engine you test, for two to four weeks.
4. Randomly assign half the pages to treatment.
5. Make one change to the treatment pages only, and request recrawling where you can.
6. Keep measuring both groups with the same questions, engines and settings for four to eight weeks.
7. Compare the change in each group using difference-in-differences.

Split-page testing is standard practice in SEO testing, and it transfers well to AI visibility because the unit of retrieval is the page.

How do you measure AI visibility changes against a control group?

A good control group is similar to the treatment group before the change:

• the same page type, topic area and market;
• similar baseline citation or mention rates;
• similar trends during the baseline period, which is the key assumption behind difference-in-differences;
• not affected by the change indirectly, for example through internal links from treated pages.

Check the baseline trends before starting. If the control group was already moving differently, the comparison will be biased. With few units, match pairs of similar pages and randomise one page in each pair to treatment.

How do I calculate difference-in-differences for AI citations?

Difference-in-differences compares the change in the treatment group with the change in the control group:

Effect = (treatment after − treatment before) − (control after − control before)

A worked example with citation rates across repeated runs:

The control group rose by 4 points without any change, perhaps because of a model update or seasonal interest. So only 8 of the treatment group's 12 points can be attributed to the change.

For the uncertainty, work at the level of questions or pages, not individual answers, because repeated runs of the same question are correlated:

1. For each question, compute its citation rate before and after, then its change.
2. Compute the average change in the treatment group and in the control group.
3. Compare the two sets of changes with a two-sample t-test, or a regression of change on group, and report the difference with a confidence interval.

This treats each question as one observation, which is honest about how much independent information you have.

How do I calculate a minimum detectable lift in AI mentions?

Before running a test, check whether it can detect the effect you care about. For comparing two rates with a 5% significance level and 80% power, a common approximation is:

Minimum detectable difference ≈ 2.8 × √(2 × p × (1 − p) ÷ n)

Here p is the baseline rate and n is the number of independent observations per group. With p = 30% and n = 300 independent observations per group, the minimum detectable difference is about 10.5 percentage points. A difference-in-differences design, which compares changes rather than levels, needs roughly 1.4 times that, about 15 points, if before and after samples are independent.

The catch is "independent". Repeated runs of the same question are strongly correlated: a brand that is named on one run is usually named on the next. So 50 questions run three times is not 150 independent observations; it is much closer to 50. At n = 50, the same formula gives a minimum detectable difference of about 26 points. In practice:

• Add questions before adding runs. More distinct questions raise independent information faster than more repeats.
• Use three to five runs per question to estimate each question's rate, then analyse at question level.
• Expect to detect only large effects with small tests. Changes that move citation rates by a few points need hundreds of questions.

Should you test one intervention or bundled SEO changes?

It depends on the goal:

• Learning what works: change one thing at a time, such as the opening paragraph, adding prices or adding schema. Bundled changes cannot tell you which part mattered.
• Improving results fast: bundle the changes you already believe in and test the bundle against a control. You learn whether the package works, not which part.
• Both: run a bundle on most pages for impact, and single-change tests on smaller groups to learn.

Tests that try several changes in combination need far more pages than most sites have. Keep designs simple.

How do you test whether changing only schema affects AI visibility?

1. Choose pages whose visible content will not change during the test.
2. Randomly add or correct structured data on half of them, server-rendered and matching the visible content.
3. Change nothing else: no content edits, internal links or new backlinks for either group.
4. Measure citations in Google's AI features, using Search Console's Generative AI performance report by page, and in assistants with repeated question runs.
5. Compare with difference-in-differences.

Expect small or no effects in assistants: tests have found that ChatGPT, Claude, Perplexity and Gemini read visible text, not JSON-LD, when fetching pages. Any effect is more plausible in Google's and Bing's own features, which parse markup when indexing.

How do you run GEO experiments on a new website?

New sites are hard to test on. Baselines are near zero, indexing takes time, and early visibility is driven by discovery rather than your changes. Useful approaches:

• Test discovery itself: how long each engine takes to find and cite new pages, and which steps speed it up.
• Use unique, truthful facts that exist only on your site, such as your own data, and measure whether and when engines cite them.
• Wait for a stable baseline before testing content changes; otherwise growth from indexing will look like an effect.
• Keep a control group of pages even on a small site, published at the same time and left unchanged.

Do not publish false or invented information to test engines. It can spread into answers and damage trust in the site.

How do we measure whether a funding announcement changes AI search visibility for our company?

You cannot randomise an announcement, so use an event study:

1. Before the announcement: track brand questions ("What is [company]?", "Is [company] reliable?") and category questions weekly for four to eight weeks, alongside the same questions for three to five comparable competitors.
2. After the announcement: keep the same schedule for eight to twelve weeks.
3. Compare your change with the competitors' change over the same period, as a comparison group.
4. Check the sources: are new citations coming from coverage of the announcement?
5. Separate the effects: search-based answers may change within days; answers from model memory may not change for months.

Look at description as well as frequency. The announcement may change how engines describe you, for example "a fast-growing company backed by…", before it changes how often you are recommended.

What mistakes invalidate GEO experiments?

• Changing the question set mid-test. New or reworded questions make before and after incomparable. Freeze the set before the baseline.
• Changing settings. Switching engine mode, model, location or logged-in state between periods creates changes of its own.
• Contaminating the control. Linking from treated pages to control pages, or running PR that mentions both, spreads the effect.
• Stopping when the numbers look good. Checking results daily and stopping at the first favourable moment inflates false positives. Fix the end date in advance.
• Counting answers instead of questions. Treating every run as independent makes small effects look significant.
• Ignoring recrawl timing. Measuring before engines have fetched the changed pages records no effect, and a later jump gets blamed on something else. Check server logs for crawler visits to the treated pages.
• Reporting only the metric that moved. Decide the main metric beforehand, and report the others as secondary.

A test with a clear hypothesis, a frozen question set, a randomised control and a fixed end date is worth more than months of loosely tracked changes.

What do people ask about GEO experiments?

How long should a GEO test run?

Usually four to eight weeks after the change goes live, after a baseline of two to four weeks. Shorter tests mostly measure recrawl timing.

Can I test across different AI engines at once?

Yes, but analyse each engine separately. They use different indexes and may respond differently to the same change.

What if the control group improves too?

That is what the control is for. It shows the change that would have happened anyway, which difference-in-differences subtracts.

Do I need a statistician?

For a simple split-page test analysed at question level, a spreadsheet and a t-test are enough. For complex designs, get help.