Human Evaluation vs LLM-as-a-Judge for Measuring Whether AI Recommends Your Business

How do the three methods compare?

Which job should each method do?

• Mention detection: rules, with help. A list of your brand name, product names, common misspellings and former names catches most mentions. Send ambiguous cases, such as a brand name that is also a common word, to the LLM judge.
• Stance and recommendation: the LLM judge. Was the business recommended, listed, compared unfavourably or warned against? At what rank? For which need? This is where the volume is, and where a rubric-following model is most useful. The labels themselves are covered in whether a negative mention counts as AI visibility.
• Factual accuracy: people, or a judge with a fact sheet. Whether an answer states your price, service area or features correctly depends on knowledge the judge does not have. Give it a current fact sheet and have people review everything it flags.
• The rubric, the gold set and audits: people. Only people can decide what "recommended" should mean for your business, and only people can tell you whether the judge is right.

What does research say about LLM judges?

The best-known study, by Zheng and colleagues in 2023, compared model judges with human preferences on two benchmarks. The strongest judge, GPT-4, reached over 80% agreement with human preferences, about the same as the agreement between humans. The same study documented biases:

• Position bias: favouring an answer because of where it appears. When the order of two answers was swapped, even the best judge gave consistent verdicts in only 65% of cases in one test.
• Verbosity bias: favouring longer answers. In a test that padded answers with repetitive lists, two of three judges were fooled 91.3% of the time, while GPT-4 was fooled 8.7% of the time.
• Self-enhancement bias: two of the three judges tested gave their own model's answers a higher win rate, though the authors called the evidence limited.
• Limited reasoning: judges misgraded math questions, even basic ones they could solve on their own, unless given a reference answer. For your use, that means giving the judge your fact sheet rather than trusting it to know.

Those tests used 2023 models and a different task, comparing two answers to the same question. Newer models are generally stronger judges, but the lesson carries over. Biases are real, they vary by model, and you find them only by testing on your own material.

How do you calibrate an LLM judge?

1. Write a codebook. Define each label in a sentence, with two or three real examples, including the hard cases: comparisons, caveats, lists without a verdict, and answers that recommend you with wrong facts.
2. Build a gold set. Pick 150 to 200 answers covering different assistants, question types and labels, including rare ones. Have two people label them independently.
3. Measure human agreement. Report both the percentage of matching labels and Cohen's kappa, which corrects for agreement by chance. Kappa matters when one label dominates. If 80% of answers do not mention you, a reviewer who always says "absent" will still agree with a careful reviewer 80% of the time without judging anything; Cohen's kappa for that pair is zero. Resolve disagreements and tighten the codebook until people agree well, since a judge cannot beat an unclear rubric.
4. Run the judge on the gold set. Give it the codebook and examples, fix the model version and settings, and ask it to quote the words in the answer that support its label. Quoted evidence makes errors easy to spot.
5. Compare with the gold labels, label by label. Overall agreement can hide a weak label. A judge that is 95% right on "absent" and 60% right on "recommended" is not ready.
6. Fix and repeat. Most errors trace back to the rubric. Typical fixes are defining "recommended" as requiring explicit preference language, or saying how to treat a list introduced as "top picks".
7. Set a bar. Aim for the judge to agree with the gold set at least as well as your two human reviewers agreed with each other.

Which bias tests are worth running?

• Name swap. Replace your brand name with a competitor's in a sample of answers and rerun the judge. The label should stay the same. If it changes, the judge is reacting to the name, not the text.
• Order swap. In list answers, move your brand from first to third place without changing what is said about it. Only the rank label should change.
• Length padding. Add neutral filler to an answer. A judge that becomes more positive is showing verbosity bias.
• Self-judging. If the judge model is from the same family as the assistant being measured, compare its labels with a judge from another family on a sample.
• Repeat runs. Run the judge twice on the same answers. Labels that change between runs mark unstable rubric areas.

How do you keep results comparable over time?

• Version everything: the codebook, the judge prompt, the judge model and its settings. Results are comparable only within one version.
• Recalibrate when anything changes. Rerun the gold set whenever you change the judge model or the rubric, or when assistants change their answer format, such as adding shopping cards.
• Audit every cycle. Have a person review a random 5% to 10% of judged answers, plus everything the judge marked low-confidence or could not quote evidence for.
• Report agreement with results. A line such as "judge agreement with human gold set: 91%" tells readers how far to trust the numbers.

When buying a visibility tool, ask how it decides that an answer "recommends" you, whether that logic has been checked against human labels, and what the agreement was. Many tools count mentions or cited sources well but say little about how they judge recommendation.

What might an illustrative case look like?

Consider an illustrative home services company tracking 60 questions across four assistants. Its first dashboard used text matching and reported a 52% "recommendation rate", which was really a mention rate. The team wrote a six-label codebook, and two marketers labelled 200 answers, agreeing on 87% of them. An LLM judge given the codebook matched the gold labels 78% of the time. Most errors came from treating any list as a recommendation. After the codebook defined "recommended" as named as a top choice or with explicit preference language, agreement rose to 89%. A name-swap test showed no change in labels. The company's true recommendation rate was 21%, a number the team could now defend.

What do people ask about measuring AI recommendations?

Can we skip human labelling if the judge model is strong?

No. Without a human gold set, you cannot know whether the judge's idea of "recommended" matches yours. The labelling is a one-off cost that is small next to the value of trustworthy numbers.

How many answers should people audit each cycle?

A random 5% to 10%, plus every low-confidence label, is a practical starting point. Increase it if audits find errors.

Should we use the same model to generate and judge answers?

It is better to judge with a different model family, or at least to check a sample against one, because judges can favour their own model's answers.