We audited our own AI visibility index. It did not survive.

In June we published a study of how much AI engines disagree about the best software. Re-examining it in September, we could neither reproduce it nor interpret its headline number. We asked each engine once, which means we never measured how much a single engine disagrees with itself. Below: three measurements that should have come first, and the four-line disclosure standard we now hold ourselves to.

The correction: what we published, and why we withdrew the headline

In June 2026 we published the AI Search Disagreement Index, reporting that AI engines fully agreed on a single top tool in 0 of 16 software categories. We presented that as a finding about the engines.

It was not sound, and the reason is simple enough that we should have caught it: we asked each engine once. A measurement taken once cannot be separated from the instrument's own noise. Before you can claim two engines disagree, you have to know how much one engine disagrees with itself across repeated asks. We never established that floor, so the number was uninterpretable rather than wrong.

We are publishing the correction here rather than editing the original quietly, and the original now carries a link to this page.

Finding 1: a model does not reproduce its own answer

We asked one model the same question repeatedly within a single session and measured how much of its own answer came back. We call this SA@n, self-agreement at n runs, reported as the overlap between the sets of company names returned.

Temperature explains most of the instability, and almost no AI-visibility product discloses the temperature it queries at. But note that temperature 0 does not reach 1.0 either. On batched mixture-of-experts inference, expert routing and floating-point reduction are not deterministic, so a provider's "deterministic" mode is not deterministic in the sense a researcher means.

Self-agreement for every model we could measure

Bootstrap confidence intervals over questions, not over pairs. Any model with fewer than 5 questions carrying repeats is reported as insufficient data rather than given a number.

These are the floors. Every claim about two models differing has to clear them.

Finding 2: model size moves the answer as much as changing lab

The same weights family at two sizes, with the same prompt, the same temperature and the same hour, set against models from entirely different labs:

The smaller model answers less often, and is more consistent when it does

openai-gpt-oss 20B was the only model in this run that returned an empty answer to some questions. Re-asked directly, it returns nothing at all for them, so it is declining rather than being mis-parsed. It answered 27 of the 40 questions openai-gpt-oss 120B answered.

That matters, because a model which only answers what it finds easy would look more self-consistent for a reason that has nothing to do with consistency. So we checked, expecting to have to withdraw the result:

The gap is wider like-for-like, not narrower. And on the questions the smaller model skipped, the larger one scored 0.862, above its own average of 0.823: the skipped set was the easier one for the larger model, so the selection effect runs against this finding rather than producing it. We report the refusal rate alongside the agreement figure because "answers less often, more consistently when it does" is the complete claim, and either half alone is misleading.

Finding 3: the instruments retire faster than the studies age

We probed every model this estate has actually run in a published or frozen artifact, with a positive control in each run so that a dead API key cannot be mistaken for a retired model.

5 of 9 (56%) no longer answer. Retired at their provider: llama-3.3-70b, llama-4-scout, qwen-3.6-27b, compound, compound-mini. This is why the June index cannot be re-run at any price: most of its panel no longer exists. Every published AI-visibility benchmark carries this expiry and none of them print it.

The standard we should have used

Four disclosures. Short enough to adopt, specific enough to fail:

1. Model and size, not just the vendor name.
2. Temperature.
3. Run count per question.
4. Measured SA@n for that model, in that run.

A visibility score without a floor is a guess with a decimal point. Ours are published above, including the ones that make our own earlier work look worse. If you publish AI-visibility numbers, we would rather you adopted these four lines than cited us.

Method, data and limits

Prior work and primary sources: the Generative Engine Optimization paper (Aggarwal et al., KDD 2024) established the benchmark format this field builds on; Bing Webmaster Tools AI Performance is the first-party citation telemetry we compare against; Cloudflare Radar's crawl-to-refer ratio is the model for publishing a metric others can adopt; the KDD proceedings record carries the peer-reviewed version; and the full data release is licensed CC BY 4.0.

Questions