What is a confidence band?
A confidence band is the range around a measured score that reflects how much of the number is sampling noise rather than signal.
Language models sample from a probability distribution, so asking one question several times gives several answers. The band is calculated from how much those runs varied, as 1.96 times the standard deviation divided by the square root of the number of runs, giving an interval that should contain the true value 19 times out of 20.
Its practical use is deciding whether a change is real. If Monday's score and Friday's score have overlapping bands, nothing happened, and any explanation you attach to the difference is a story about noise.
A visibility score reported without one is not wrong so much as unfalsifiable: there is no way to tell its movement from its error.
Related
- AI visibilityAI visibility is how often, and how prominently, a brand is named by AI assistants when people ask questions in its category.
- Sampling varianceSampling variance is the run-to-run variation you get from asking a language model the same question more than once.
- Model driftModel drift is when an engine's answers change because the provider changed the model behind it, not because anything about your brand changed.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.