Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — 2026-10-31
Artificial Analysis released Intelligence Index v5, shifting 60% of scoring to private tests and widening the gap between proprietary and open-weight models to 12 points. Meanwhile, OpenRouter data shows Chinese models dominating token usage, with DeepSeek V4.1-Flash seeing an 801% weekly surge.
Evals and Leaderboards: LMArena, SWE-bench, ARC-AGI — 2026-10-31
Top developments
Artificial Analysis v5 Overhaul Reshapes Rankings
Artificial Analysis released its Intelligence Index v5, a major update that moves 60% of the composite score to private test sets to combat benchmark contamination. This methodology change significantly altered the competitive landscape, widening the performance gap between proprietary models (like Claude, GPT, and Gemini) and open-weight models from 3 points to 12 points. The update also introduced new coding and science tasks, resulting in no single model leading all sub-categories.
Kimi K3.1 Pro Falls in Global Rankings
The update to major benchmarks has dealt a significant blow to Chinese contender Kimi K3.1 Pro, which is no longer ranked as the fourth-best AI model globally. This shift highlights the impact of stricter, private evaluation standards on models that previously excelled on public, potentially contaminated leaderboards.
AA-Omniscience v1.1 Targets Model Guessing
Artificial Analysis also released AA-Omniscience v1.1, a benchmark designed to detect when AI models guess rather than abstain from answering. The new version shrinks the dataset to 3,000 questions across 6 domains, removing items that were saturated or ambiguous to better measure true knowledge boundaries.
OpenRouter Usage Dominated by Chinese Models
While proprietary models lead in capability scores, usage metrics tell a different story. OpenRouter data for October 19–25 shows that the top five models by token volume were all Chinese. DeepSeek V4.1-Flash-1017 led the pack with 33.6 trillion tokens, marking an 801% week-over-week increase. Chinese models collectively accounted for 93.54 trillion tokens in that week, continuing a 26-week streak of dominance in raw volume.
Local view
In China, SuperCLUE released a new Image-to-Video (I2V) model leaderboard where MiniMax H3 secured the top domestic spot with a total score of 71.46, including a high application capability score of 75.82. Additionally, Zhihu reports that DeepSeek added new agent models and GPT updated its agent models in the week of October 26–30, reflecting rapid iteration in the Chinese open-source community.
Context & numbers
Cost efficiency remains a key differentiator in the new Artificial Analysis Index. On the v5.0 index, GPT-6.1 Terra scored 51.0 with a cost of $0.96 per task, while GPT-6.1 Sol scored 54.5 at $1.27 per task. The total token volume on OpenRouter for the recent week was 188 trillion, a 14.63% increase.
On the radar
- Benchmark Contamination Studies: A new taxonomy paper on benchmark contamination organized by defeated mitigation strategies was released in late August but continues to influence methodology discussions in October.
- Epoch AI Data Update: Epoch AI updated its benchmark results database on October 30, 2026, providing fresh external data points for leading models.
This content was collected, curated, and summarized entirely by AI — including how and what to gather. It may contain inaccuracies. Crew does not guarantee the accuracy of any information presented here. Always verify facts on your own before acting on them. Crew assumes no legal liability for any consequences arising from reliance on this content.
Powered by
