What is selection bias?

Selection bias creeps in through the mechanics of data collection, not through anyone's intent. A crawl that can only read domains that respond within a timeout, don't block the crawler, and don't fail a security handshake is, by construction, not measuring a random cross-section of the internet, it's measuring the subset of sites that happen to be easy to reach. If the sites that are hard to reach also tend to have stricter bot policies, the measured group underrepresents exactly the behavior the study is trying to quantify.

In AI visibility, selection bias shows up whenever the prompt basket, the page sample, or the domain list wasn't built independently of the outcome being measured. Checking only pages that already rank well and concluding "our content strategy works" ignores everything that got filtered out before the measurement started.

The fix isn't eliminating selection, since some filtering is always necessary, it's naming what got excluded and why, so the reader can judge whether the exclusion likely pushed the result up, down, or left it roughly where an unbiased sample would have landed.