FanBell: this scan vs. the previous one

No change was detected: 2 of 39 questions gained the majority verdict and 3 lost it (exact McNemar p = 1.00; -3 points, question-resampling sensitivity range -13 to +8 points). This is not the same as showing that no change occurred; this scan may miss small or inconsistent movements. With 39 matched questions, this comparison has limited power and is mainly able to detect large, directionally consistent changes. Under the planning assumptions used here — 80% power, a two-sided 5% test and an assumed 20% of questions disagreeing between the two scans — a change of about 20 points has about 80% power in one primary comparison. This is not a cutoff: smaller changes can sometimes be detected, and larger changes are not guaranteed to be detected.

The verdict is an exact McNemar test on the 39 question(s) both scans asked, comparing each question's majority verdict — not the rates, and not the flip counts below. A permutation test over the graded per-question fractions gives p = 1.000. This result was adjusted with 1 per-engine comparison, for 2 comparisons total, using Benjamini–Hochberg; q = 1.000.

Only the two disagreeing cells carry information: 3 question(s) lost the majority verdict, 2 gained it — exact McNemar p = 1.000 (mid-p 0.688). Effect -3 points, question-resampling sensitivity range -13 to +8 points, over questions.

The primary comparison specified in the current method version is an exact two-sided McNemar test on matched question-level majority verdicts. α = 0.05, Benjamini–Hochberg across 2 comparison(s). Every range here is a question-resampling sensitivity range: resampling which of the frozen questions you look at moves the answer by this much. It is not a calibrated re-test range and not a market confidence interval, and the bootstrap’s own simulated coverage across the designs we ship (scripts/ci_coverage_sim.py) runs 0.90–1.00, so no 95% level is claimed for it.

A detected change does not show what caused it: your site may be unchanged while models, indexes, retrieval systems or other external conditions move. This is measurement evidence, not attribution.

By engine, each corrected in the same family (the engines are separate populations): OpenAI no change detected — 2 gained / 3 lost of 39, p = 1.000.