What is statistical significance?

When a citation rate moves from one measurement to the next, the question worth asking is whether that move could plausibly have happened by chance, given the size of the sample, even if nothing real changed. Statistical significance is the label applied when the answer is no, the move is bigger than chance alone would typically produce. It is a statement about the sampling process, not a statement about how much the move matters.

The common misread is treating "significant" as a synonym for "big" or "worth acting on." Run a large enough prompt basket and even a one-point shift in citation rate can clear the bar for significance, because a bigger sample makes it easier to detect small, real differences. That doesn't make the one point commercially meaningful. The reverse also happens: a large, real shift can fail to reach significance in a small sample simply because the sample wasn't big enough to tell it apart from noise.

Significance answers "is this probably not zero," and stops there. Whether it's worth a strategy change is a separate question about size and cost, one the statistics don't answer for you.