How to Measure Performance Across Multi-Turn AI Conversations and Product Recommendations
To measure performance across multi-turn AI conversations, script the paths real buyers take and track your brand through every turn, not just the first answer. Buyers ask for options, add constraints such as budget or size, compare two or three choices, raise objections and finally ask which to buy. A brand can appear in the first list and vanish at the second question, or appear only once a specific need is mentioned. Useful measures are entry rate, survival rate through each turn and final recommendation rate. Add the turn where you drop out and why, and the accuracy of what is said about you. Run each path several times, because answers vary. For products, track which specific model is recommended and whether its price and availability are right.
Why isn't a single prompt enough?
- Conversations narrow. The first answer is a broad list; follow-ups remove options that do not fit the buyer's constraints.
- The final pick matters most. Research on AI-assisted shopping has found that brands named as recommendations influence behaviour far more than brands merely mentioned.
- Order effects exist. Research on how language models choose among options found they can favour options by position, which makes the path to the final choice important.
- Different questions reveal different weaknesses. You may be named for "best options" but dropped at "which is cheapest to run?" because your running costs are not published.
How do you design conversation scripts?
- Define buyer types: for example, a small agency owner on a budget, an enterprise IT lead with security needs, a parent buying for a child.
- Write an opening question for each, in the words that buyer would use.
- Add follow-ups that reflect real decision steps: a constraint ("under £500"), a comparison ("how does A compare with B?"), an objection ("isn't B hard to set up?") and a decision ("which one should I buy?").
- Plan branches. If your brand appears, follow up about it; if not, ask "what about [your brand]?" to see how the assistant describes you when prompted.
- Keep scripts fixed so results can be compared over time.
Most programmes start with five to ten paths per product line, each with four to six turns.
Which metrics should you track?
| Metric | Definition |
|---|---|
| Entry rate | Share of conversations in which your brand is named at any turn |
| First-turn mention rate | Share in which you are named in the first answer |
| Survival rate by turn | Of conversations where you appeared earlier, the share where you are still under consideration at turn N |
| Final recommendation rate | Share in which you are the recommended choice at the decision turn |
| Drop-off turn and reason | The follow-up at which you are removed, and the stated reason, such as price or a missing feature |
| Late entry rate | Share where you are absent at first but named after a specific constraint |
| Claim accuracy | Share of statements about your brand that are correct across all turns |
| Cited sources by turn | Which sources support the answer at each step |
Report these by buyer type and engine. Averages across very different paths hide the useful detail.
How many runs, and with what settings?
Run each path at least five to ten times per engine per period, because answers vary between runs. Use fresh sessions with memory and personalisation off, fixed location and language, and record the model and whether search was used. Results from an assistant's API can differ from its consumer app, so do not mix them in one comparison. That difference is covered in why ChatGPT API answers differ from the web app when tracking citations.
How do you score conversations at scale?
- Extract brands and products from each turn with a dictionary of names and variants, including misspellings and model numbers.
- Label each turn: named, recommended, rejected, with the reason where given.
- Check claims about your brand against a fact sheet.
- Use a language model as a judge for labelling large volumes, but validate it against human-labelled samples and review disagreements.
- Store full transcripts so you can recheck labels later.
Counting methods and confidence intervals are covered in how to count brand mentions in AI responses with keyword regex and confidence intervals.
How do you measure product-level recommendations?
For online stores and manufacturers, track the specific product, not just the brand:
- Which model or SKU is recommended, and whether it is current.
- Price and availability accuracy, against your live data.
- Which merchant is linked when assistants show product cards or buying options.
- Attributes quoted, such as size, capacity or compatibility, and whether they are right.
Shopping features in assistants often show product cards built from merchant feeds, so feed accuracy is part of the measurement. Tracking at product level is covered in how to track product visibility by SKU in LLM answers.
How do you compare against competitors?
Track the same metrics for your main competitors in every conversation. The most useful comparison is share of final recommendations: of all conversations that end with a clear pick, how often each brand is chosen. Also note head-to-head turns, where the buyer asks the assistant to compare you with a named rival, and record which brand the assistant favours and why. A competitor that consistently survives a follow-up you fail usually has clearer evidence for that requirement, which shows you exactly what to publish or earn.
How do you turn results into actions?
Plot a funnel for each path: named, still considered after each follow-up, recommended. The step where the funnel drops sharply tells you what to fix:
- Dropped at a price question: publish clear pricing and total-cost information.
- Dropped at a feature or compatibility question: state that capability explicitly, with specifics.
- Dropped at an objection: publish honest answers to the objection, such as set-up time with examples.
- Named only when prompted: work on third-party evidence, since the assistant knows you but does not rank you among the default choices.
Rerun the same paths after changes, and compare with paths you did not work on.
What might an illustrative case look like?
Consider an illustrative robot vacuum brand that tracks six conversation paths across four assistants, ten runs each. Its first-turn mention rate is 55%, but its final recommendation rate is only 8%. The funnel shows most drop-offs at the turn "Which is best for pet hair?". The assistants say the brand's models "may struggle with long hair", citing an old review of a previous model. The brand publishes a page with pet hair test details for its current models, updates its product feeds and lends its current models to two independent testing sites that already include a pet-hair test, with no conditions on what they publish. Three months later, survival past the pet hair turn rises from 20% to 58%, and the final recommendation rate reaches 21%.
What do people ask about measuring AI conversations?
How many turns should a script have?
Four to six covers most buying journeys. Longer scripts add cost without much new information.
Can we automate multi-turn testing?
Yes, through APIs or browser automation, but check a sample in the consumer apps, since answers can differ.
Should we let an AI simulate the buyer's follow-ups?
It can explore paths you have not thought of, but fixed scripts are needed for comparisons over time. Use both.
How often should we run conversation tests?
Monthly for core paths, and after significant changes to your pages, prices or products.
Published by The Kovmere Team · updated Sep 29, 2026 · More resources