What is a benchmark?

A benchmark is meant to hold still so everything else can be measured against it, a fixed set of prompts, a fixed scoring method, run repeatedly so results from different dates or different tools can be compared on equal footing. The value of a benchmark comes entirely from that stability.

The problem specific to AI is that the systems being benchmarked don't hold still even when their name doesn't change. A model provider can update the model behind a given product name at any time, without announcing it, which means a benchmark score from three months ago may describe a system that no longer exists in that form. Treating an old benchmark result as current information about today's model is a common and easy mistake, especially since the naming stays the same across the update.

A benchmark result is only informative alongside the date it was run and, where possible, the model version it was run against. Without that, a benchmark comparison across time is really a comparison between two different, unlabeled systems.