Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — 2026-10-28
Artificial Analysis released Intelligence Index v5, introducing private datasets to combat benchmark contamination and revealing a 13x cost disparity among top models. In China, DeepSeek V4.1-Flash-1017 saw an 801% surge in weekly token usage, while MiniMax H3 claimed the top spot in the SuperCLUE video generation leaderboard. Meanwhile, new research highlights the fragility of leaderboard rankings, with one model dropping from first to 69th due to evaluation methodology changes.
Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — 2026-10-28
Top developments
Artificial Analysis Launches Index v5 with Private Data
Artificial Analysis announced the release of Intelligence Index v5, a significant update featuring two new benchmarks and three upgrades. For the first time, the majority of the Index is built on private datasets, a strategic move to mitigate data contamination and "gaming" of public benchmarks. This shift addresses growing concerns that public test sets are increasingly polluted, leading to inflated scores that do not reflect real-world capability.
Cost Variance Reveals Efficiency Gaps
The new Index highlights a stark economic divide among frontier models. Among the top 10 highest-scoring models, cost per task varies by more than 13x. GPT-6.1 Terra (max) is the most cost-efficient at $0.96 per task, while Claude Fable 5.1 (max) is the most expensive at $13.04. This data provides enterprises with critical insights for procurement, moving beyond raw capability scores to include cost-performance ratios.
DeepSeek's Surge in OpenRouter Usage
In the Chinese market, DeepSeek V4.1-Flash-1017 experienced an explosive 801% increase in weekly token usage, reaching 33.6 trillion tokens on the OpenRouter platform between October 19 and 25. This surge contributed to Chinese models dominating the top five positions, with total domestic model usage hitting 93.54 trillion tokens for the week. This reflects a growing preference for high-efficiency open-weight models in cost-sensitive markets.
Leaderboard Volatility and Methodology Critiques
A recent analysis by ORCARouter highlighted the fragility of crowd-sourced arenas like LMArena, noting that a model can fall from first to 69th place without any change in its underlying weights, purely due to shifts in user preference or evaluation criteria. The article argues that crowd arenas measure preference rather than objective capability, urging developers to cross-reference multiple sources.
Local view
Chinese Media Focus on Video and Token Efficiency Chinese tech media is closely tracking the "domestic model dominance" narrative. Sina News reported that Chinese large models have topped the OpenRouter leaderboard for 26 consecutive weeks, driven by the massive token volume of DeepSeek and other local players. Additionally, Sohu reported that MiniMax H3 secured the first place in the SuperCLUE Image-to-Video (I2V) model leaderboard with a total score of 71.46, highlighting the intense competition in multimodal capabilities beyond just text.
Context & numbers
- OpenRouter Weekly Volume: 188 trillion tokens, a 14.63% week-over-week increase.
- Cost Per Task: $0.96 (GPT-6.1 Terra) to $13.04 (Claude Fable 5.1) for top-tier models.
- ARC-AGI Scores: GPT-6 Astra currently leads the ARC-AGI leaderboard with a score of 0.985, while leading the ARC-AGI-3 variant with 62.7.
- SWE-bench Verified: Claude Fable 5 leads with a score of 0.950, solving 500 software engineering problems.
On the radar
- Epoch AI Updates: Epoch AI updated its benchmark database and Capabilities Index (ECI) on October 28, 2026, providing independent, vendor-free reference data for enterprise teams.
- OWASP Agent Control Standard: The Chinese-language tech community is discussing the implications of the OWASP Agent Control Standard v0.1 released in September, which aims to standardize security protocols for AI agents, potentially influencing future agent-specific benchmarks.
- Benchmark Contamination Research: A new paper, "Benchmark Leakage Trap," explores how data leakage impacts LLM-based recommendation systems, suggesting that domain-irrelevant leakage can degrade accuracy, complicating the interpretation of scores in specialized domains.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
