He Wants AI Benchmarking To Be a Science, Not a Dark Art

Benchmarks are the rulers of AI, in both senses of the word. As measuring sticks, they work according to an extremely simple logic: If you want to know what an AI model can do, devise a standardized test to gauge it. But AI benchmarks can also dominate the behavior of the entire field. In 1987, the…

Benchmarks are the rulers of AI, in both senses of the word. As measuring sticks, they work according to an extremely simple logic: If you want to know what an AI model can do, devise a standardized test to gauge it. But AI benchmarks can also dominate the behavior of the entire field. In 1987, the Defense Advanced Research Projects Agency’s first benchmark on automatic speech-to-text… Source

Source: Quanta Magazine — Published October 16, 2026 at 13:25 UTC — Category: Research