How to Count Brand Mentions in AI Responses With Keyword Regex and Confidence Intervals

Why use regex rather than a tool's summary figure?

Visibility tools report mention rates, but you often cannot see how they match names or how many answers sit behind a number. Counting from raw text with your own patterns gives you:

• Control over what counts, such as product names, former names and misspellings.
• Fewer false matches, by excluding common words and unrelated companies with similar names.
• Reproducibility. Anyone can rerun the count on the stored text.
• Honest uncertainty, because you know the sample size behind every rate.

How do you build the brand dictionary?

List every way assistants might refer to you:

• Official name and short forms: "Acme Analytics", "Acme".
• Product names and model numbers: "Acme Insight", "Insight Pro".
• Former names after a rebrand.
• Domain names: "acmeanalytics.com".
• Common misspellings and spacing variants: "AcmeAnalytics", "Acme-Analytics".
• Possessives and plurals: "Acme's".
• Other languages and scripts for international tracking.

Keep a separate exclusion list for false matches, such as unrelated companies or common words that share your name.

How do you write the patterns?

Use word boundaries and case-insensitive matching, and treat each dictionary entry as a literal string. In Python, a simple version looks like this:

import re
names = ["Acme Analytics", "Acme Insight", "acmeanalytics.com"]
pattern = re.compile(r"\b(" + "|".join(re.escape(n) for n in names) + r")\b", re.IGNORECASE)
mentioned = bool(pattern.search(answer_text))

A few rules prevent common errors:

• Match whole words. Without word boundaries, a short brand name matches inside longer words.
• Handle ambiguous names separately. If your brand is a common word, require a nearby product or category term, or review matches by hand.
• Separate text from citations. Count mentions in the answer text and domains in the cited links as two different measures. An answer can cite your site without naming you, or name you without citing you.
• Ignore echoes of the question. If the prompt contains your brand, exclude the repeated question from the answer text, or track branded prompts separately.

How do you calculate a confidence interval?

For a mention rate, use the Wilson score interval, which behaves well with small samples and rates near 0% or 100%. With x answers mentioning you out of n, p = x / n and z = 1.96 for 95% confidence:

• centre = (p + z² / 2n) / (1 + z² / n)
• half-width = z × √(p(1 − p) / n + z² / 4n²) / (1 + z² / n)
• interval = centre ± half-width

Worked examples at a 30% mention rate:

The interval narrows roughly with the square root of the sample size. Halving the width takes about four times as many answers.

How do you handle repeated runs of the same prompt?

Runs of the same prompt are not independent. If one prompt almost always mentions you and another almost never does, pooling 200 answers from 20 prompts overstates your precision. Two practical approaches:

• Calculate a rate for each prompt, then average across prompts. Report the spread between prompts as well as the average.
• Bootstrap by prompt. Resample whole prompts with replacement many times, recalculate the overall rate each time, and use the middle 95% of results as the interval.

Both give more honest uncertainty than treating every answer as a separate observation.

How do you tell whether a change is real?

Compare intervals and sample sizes, not just rates. If this month's interval sits clearly above last month's, the change is probably real. If they overlap heavily, it may not be. For a formal check, a two-proportion test works for simple samples, and bootstrapping by prompt works for repeated runs. Also compare with control prompts you did not work on, since the engines themselves change. How to set up before-and-after comparisons is covered in how to retest the same AI prompts after updating your website.

How should you report rates across engines and question groups?

Report each engine and each question group separately, with its own sample size and interval, before showing any overall figure. Engines differ in how often they name brands, and question groups differ in difficulty, so a combined rate can hide a fall in one engine behind a rise in another. When you do combine, weight each engine or group deliberately, for example by how much of your audience uses it, and say so. Show competitors' rates with the same method beside yours, since relative position is often more meaningful than the absolute rate.

What should you store?

For each answer:

• prompt, engine, model, date, settings and run number;
• full answer text and cited URLs;
• whether each brand pattern matched, and the matched text;
• the dictionary version used.

For each reported figure:

• rate, number of answers, number of prompts and interval;
• the method used for the interval.

Keeping the dictionary version means you can recount old answers when you add a new variant, instead of losing history.

What does regex not tell you?

A match says a brand was named, not how. Add separate labels for whether the mention is a recommendation, a neutral listing or a warning, and whether the facts stated are correct. Those labels usually need human review or a validated model judge. Measuring accuracy is covered in how to calculate a brand accuracy rate across AI engines.

What might an illustrative analysis look like?

Consider an illustrative CRM company that tracks 12 prompts, ten runs each, in one engine. It is mentioned in 36 of 120 answers, a rate of 30% with a Wilson interval of about 22% to 39%. The next month it is mentioned in 54 of 120, or 45%, with an interval of about 36% to 54%. The intervals barely overlap, and the per-prompt view shows gains in 9 of the 12 prompts, so the team treats the rise as real. A regex review also finds that 4 of the new matches were an unrelated company with a similar name. The team adds it to the exclusion list and recounts, lowering the rate to 42%.

What do people ask about counting AI brand mentions?

Should we count every occurrence or just whether the brand appears?

Whether it appears. Repeated mentions in one answer do not make it more visible to the reader in a way that is easy to compare.

How many answers do we need?

For a rough picture, about 100 per engine per period, which gives a margin of about ±9 points at a 30% rate. To tell a 5-point change from noise you need roughly 650 answers per period, and roughly 1,800 for a 3-point change, spread across as many different prompts as possible.

Can we use a language model instead of regex?

Yes, for messy cases such as ambiguous names, but validate it against hand-checked samples and keep the regex count as a baseline.

Should confidence intervals appear in client reports?

Yes. They prevent over-reading small changes and make results more credible.