Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — October 26, 2026
This week, the focus in AI evaluation shifted toward the integrity of benchmark scores and the rapid deflation of inference costs. Independent researchers like Moritz Hardt are calling for a scientific overhaul of "broken" benchmarks, while data shows the cost of achieving frontier-level capability has dropped 750-fold in 18 months. Meanwhile, Chinese models like Kimi K3.1 Pro are challenging Western dominance in specialized coding benchmarks.
Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — October 26, 2026
Cost of capability plummets 750-fold in 18 months
According to new data from BigGo Finance, the price of achieving equivalent AI capability has dropped roughly 750-fold over the past 18 months. This trend is most visible when measured against the GPQA Diamond benchmark, where OpenAI’s GPT-5.6 Luna now matches the 75% score previously achieved by GPT o3 at a fraction of the cost. The cost per question for GPT-5.6 Luna is now $0.0004, compared to $0.30 for its predecessor, signaling a dramatic shift in the economics of high-level reasoning.
Moritz Hardt argues benchmarks are "fundamentally broken"
In a recent profile by Quanta Magazine, Moritz Hardt, author of a new book on AI benchmarks, argues that standardized tests for artificial intelligence are "fundamentally broken — but not unfixable." Hardt contends that the current evaluation landscape is often a "dark art" rather than a science, with scores frequently detaching from real-world usefulness. His work emphasizes the need for rigorous methodology to distinguish between genuine capability and benchmark gaming.
Kimi K3.1 Pro overtakes GPT-6 in coding benchmark
On October 24, the Chinese model Kimi K3.1 Pro achieved a score of 65.7% on Terminal-Bench 4.0, surpassing GPT-6 Astra’s 58.2%. This marks the first time a domestic Chinese model has outperformed GPT-6 on this specific coding benchmark, highlighting the rapid progress of non-US frontier models in specialized domains like software engineering and terminal-based operations.
Goodhart’s Law resurfaces in benchmark debates
A thread by Contrary Research highlighted the relevance of Goodhart’s Law to the current AI evaluation crisis, noting that "when a measure becomes a target, it ceases to be a good measure." The discussion pointed out that as specific benchmarks like SWE-bench and LMArena become the primary KPIs for model development, they risk becoming less reliable indicators of true intelligence, leading to "hollow numbers games" and potential contamination.
Local view
Chinese media outlets are closely monitoring the shift in leaderboard dynamics, with platforms like Smzdm (What is Worth Buying) emphasizing the symbolic victory of Kimi K3.1 Pro’s 65.7% Terminal-Bench score over GPT-6. The coverage frames this not just as a technical win, but as a milestone for domestic model capability, suggesting that Chinese labs are closing the gap in specialized, agentic coding tasks.
Context & numbers
- GPQA Diamond Cost Efficiency: The cost to achieve a 75% score on GPQA Diamond dropped from $0.30 per question (GPT o3) to $0.0004 per question (GPT-5.6 Luna), a 750-fold reduction.
- Terminal-Bench 4.0 Scores: Kimi K3.1 Pro scored 65.7%, compared to GPT-6 Astra's 58.2%.
- Epoch AI Data Update: Epoch AI updated its benchmark database on October 25, 2026, providing the latest independent performance metrics for leading models.
On the radar
- Contamination Resistance: Researchers are increasingly focusing on "contamination-resistant" datasets, with new papers evaluating how well current methods prevent data leakage from KV cache representations.
- OpenAI Training Pause: Reports from Sina Finance indicate that OpenAI has resumed training its most advanced models after a one-month pause, though with restrictions on real-time internet access during the initial phase.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
