Why Artificial Intelligence Benchmarks Are Broken
A computer scientist argues that artificial intelligence benchmarks are broken because companies can easily train models to pass specific tests. However, past progress happened because model rankings remained surprisingly stable despite flawed test methods.
In short: A computer scientist argues that artificial intelligence benchmarks are broken because companies can easily train models to pass specific tests. However, past progress happened because model rankings remained surprisingly stable despite flawed test methods.
We rely on standardized tests to measure how smart artificial intelligence is, but the tools we use to grade these systems might be fundamentally broken.
What happened, in plain words
Moritz Hardt, a computer scientist and director at the Max Planck Institute for Intelligent Systems, explains in his book that machine learning benchmarks have been flawed for decades. Traditionally, developers were supposed to lock away a test set of data to fairly rank models. Instead, the field ignored this warning, placing test sets on the internet where developers could use them during training. While absolute score numbers can change dramatically with tiny tweaks to the test set, relative rankings between models stayed stable enough to drive progress by sheer luck. Today, with large language models, this accidental reliability has vanished because creators can strategically prepare their models for known tests.
Key points
- The Holdout Method Ignored: Standard textbooks advise locking away test sets to evaluate models fairly, but the machine learning field largely ignored this rule and published test data online.
- Absolute Numbers Lack Meaning: Changing a test set only slightly can cause accuracy numbers to swing wildly, meaning an absolute score like 90 percent does not accurately reflect true skill.
- Rankings Remained Stable: Historically, even though absolute scores were shaky, relative comparisons between models stayed stable, which accidentally saved machine learning from a replication crisis.
- Studying for the Test: Modern large language model creators can build data mixes specifically to promote benchmark performance, letting a worse model win by preparing for the test.
Terms explained
- Benchmark — A standardized test used to measure and compare the capabilities of different computer systems or models. Example: A spelling test given to every student in a school district to see which class spells the best.
- Training Set — The collection of data that developers feed into a machine learning model to help it learn and improve. Example: A stack of flashcards a student uses to study before taking an actual exam.
- Test Set — A separate collection of data kept specifically for evaluating how well a model performs after training. Example: The final exam questions that the teacher keeps secret until test day.
- Large Language Model — An artificial intelligence system trained on vast amounts of text to understand and generate human-like language. Example: An advanced computer program that can write essays, answer questions, and summarize long documents.
Critical analysis
What is encouraging. The work offers a refreshing and clear-eyed critique from an established computer scientist, pointing out structural flaws in how an entire field measures its own progress instead of simply hyping the latest capabilities.
What deserves caution:
- Based on expert opinion. The insights come from an interview and a book written by a single expert, rather than a broad, peer-reviewed scientific study with experimental data.
- Missing historical theory. The source notes that there is no comprehensive theory explaining why relative rankings worked so well in the past, meaning this past success is described largely as luck.
- Unclear future solutions. While the interview highlights a deep crisis in modern artificial intelligence testing, it cuts off before offering a fully tested solution to fix the problem.
Bottom line: Artificial intelligence benchmarks give misleading absolute scores, and modern models can easily study for tests to game their rankings.
What we still don't know
The source text cuts off mid-sentence while discussing a historical story about an Olympic runner, leaving the proposed solutions to the benchmarking crisis unmentioned.
Based on reporting from Quanta Magazine. This is an independent explainer, written in our own words with AI assistance; Quanta Magazine has not reviewed or endorsed it. Read the original for the full details.