What is an A/B test?

A true A/B test needs two versions running at the same time, with traffic or trials divided between them, so both experience the same external conditions. On a website this is straightforward: half your visitors see version A, half see version B, in the same week, subject to the same events. For AI visibility, you generally can't split a model's output that way. You have one page, you change it, and you measure before and after, which means the comparison is contaminated by anything else that changed in between, including a model update.

This doesn't mean before-and-after comparisons are worthless, but it means they need a control group of unchanged pages measured over the same window to approximate what a real A/B test would isolate. Calling a simple before-and-after a "test" without that control overstates how much the comparison can actually prove.

Where genuine A/B testing does work is on the input side: running two different phrasings of the same prompt basket against the same content, at the same time, to see which surfaces more citations. That splits exposure the right way, even though the output side stays uncontrolled.