What is reproducibility?
Reproducibility is whether repeating the same measurement under the same conditions gets you the same result, and for AI visibility, exact reproducibility usually isn't achievable at all.
In most measurement, reproducibility is the baseline expectation: run the same test again and you should get the same answer, and if you don't, something is wrong with the method. AI visibility measurement doesn't get to assume that. Because of nondeterminism in how models generate answers, running the identical prompt basket against the identical page twice can produce two different citation counts, with no error anywhere in the process.
This changes what a reasonable claim looks like. Instead of asking "did we reproduce the exact citation rate," the useful question is "does the result fall within the range we'd expect from re-running this," which is really a question about the margin of error and confidence band around the number, not about hitting an identical figure twice.
Reporting a single AI visibility measurement as if it were exactly reproducible, "we measured 31% citation rate," without any indication of the range that number could land in on a re-run, overstates the precision the method can actually deliver.
Related
- NondeterminismNondeterminism is the property that running the exact same input through a model can produce different output each time, and it doesn't fully go away even when settings ask the model to be as predictable as possible.
- Confidence bandA confidence band is the range around a measured score that reflects how much of the number is sampling noise rather than signal.
- Margin of errorA margin of error is the range around a measured number, such as a citation rate, within which the true value probably falls given the sample it came from.
- Sampling varianceSampling variance is the run-to-run variation you get from asking a language model the same question more than once.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.