The Great Divergence: Analyzing the Open-Weight Collapse in Artificial Analysis Index v5

For months, the prevailing narrative in the AI community has been that the gap between closed-source frontier models and open-weight alternatives is rapidly closing. However, the release of the Artificial Analysis Intelligence Index v5 on October 27, 2026, has delivered a stark correction to that optimism . The updated index reveals that the margin between the top proprietary model and the best open-weight model expanded from 3 points in version 4.3 to a 12-point gap in v5 .

This shift is not merely a result of new model releases, but a fundamental overhaul of how "intelligence" is measured. By prioritizing private datasets and agentic, multi-step reasoning over single-turn Q&A, Artificial Analysis has exposed a significant capability deficit in the open-weight ecosystem .

The War on Contamination: The Shift to Private Data

The most critical change in v5 is the aggressive move toward secrecy to combat benchmark contamination—the process where models effectively "memorize" test answers found in their training data . For the first time, 60% of the composite score is derived from private test sets or held-out answers, compared to approximately 45% in v4.3 .

To achieve this, the index now integrates several high-friction evaluations:

FrontierCode 2.0: A private benchmark from Cognition featuring 132 real-world coding tasks graded on whether a human maintainer would actually merge the pull request .
Terminal-Bench-Science: 70 agentic research tasks executed in a live terminal environment .
HLE-Diamond: A refined, 1,000-question version of Humanity's Last Exam designed to discriminate at the absolute frontier of reasoning and knowledge .

Commentary: This transition represents a pivotal moment in AI evaluation. We are moving away from "static knowledge" tests toward "procedural capability" tests. What this means in practice is that the "intelligence" being measured is no longer about what a model knows, but what it can do in a restricted, unscripted environment. The overlooked implication is that by making 60% of the index secret, Artificial Analysis has essentially become a "black box" judge. While this prevents gaming the system, it removes the community's ability to audit for errors or biases in the test sets, forcing the industry to trust the benchmarker's internal integrity over transparent, reproducible science.

The Performance Hierarchy: Proprietary Dominance

Under the new regime, proprietary models have secured a total monopoly on the top five positions . Gemini 4 Carbon (high) is the highest intelligence model with a score of 59, followed by Claude Opus 5.5 with 58 points, GPT-6 Astra (57), and GPT-6.1 Sol (55) .

However, no single model is a universal victor. The results show a fragmented landscape of specialization:

Claude Opus 5.5 dominates FrontierCode 2.0 (46%) and HLE-Diamond (68%) .
GPT-6 Astra leads in scientific agentic work via Terminal-Bench-Science (63%) and the CritPt v2 hard subset (72%) .
Gemini 4 Argon excels in SaaS workflows via AutomationBench-AA (78%) .
Muse Wonder Full takes the lead on GDP.pdf .

Commentary: The data suggests that we have reached a plateau where "general intelligence" is actually a collection of niche strengths. The fact that the leader in general knowledge is not the leader in scientific research indicates that the frontier is diversifying. For enterprises, the takeaway is clear: picking a model based on a single composite score is a mistake. Depending on whether the workload is scientific research or office automation, the "best" model changes entirely.

The Open-Weight Struggle

Open-weight models have plummeted in the rankings, primarily because they struggle with the new agentic and private evaluations . Kimi K3.1 Pro is currently the strongest open-weight contender at 46 points, though its weights were not yet released at the time of publication . Among models with weights already available for download, the situation is more dire: DeepSeek V4.1 Flash 1017 and GLM-5.3 lead the released-weight field at 32 points—a staggering 27 points behind the overall leader .

Even the smallest proprietary models, such as Claude Haiku 5.5 (36 points), now outperform the best available open-weight models . The gap is most pronounced in the hardest tasks; for instance, on Terminal-Bench-Science, GPT-6 Astra scores 63%, while Kimi K3.1 Pro reaches 34%, and most other open models fall below 10% .

Commentary: The collapse of open-weight scores in v5 suggests that prior benchmarks were inadvertently flattering these models by testing capabilities they had already "seen" in training . The widening gap highlights a massive investment divide in reinforcement learning (RL) infrastructure. Proprietary labs have clearly outpaced the open community in training models to act as autonomous agents in terminal environments. While some argue the gap will narrow as new models from Moonshot AI or Zhipu AI arrive, this v5 data serves as a cold shower for the "open-source parity" narrative.

The Economics of Effort: Cost vs. Capability

Version 5 also exposes the hidden cost of high-reasoning AI. Because agentic tasks are token-intensive—median output tokens per task rose from 35k to 59k—the cost of running these models has spiked .

There is now a dramatic variance in cost-to-performance ratios:

High-End Premium: Claude Opus 5.5 (max effort) costs $10.04 per task for a score of 58 .
Efficiency Frontier: GPT-6.1 Sol (max effort) achieves a score of 55 at just $1.27 per task .
Budget Tier: GPT-6.1 Terra is the most economical of the top ten, at $0.96 per task .

Furthermore, "effort" settings now create massive swings in performance. Claude Opus 5.5's score jumps from 35 at low effort to 58 at max effort, but at a tenfold increase in cost ($0.98 vs $10.04) .

Commentary: We have entered the era of "compute-on-demand" intelligence. The massive price difference between GPT-6.1 Sol and Claude Opus 5.5 for a mere 3-point difference in composite score suggests that the marginal utility of the absolute frontier is diminishing for most users. In practice, the most successful AI implementations will not be those using the highest-scoring model, but those that dynamically orchestrate effort levels—using "low effort" for simple tasks and reserving "max effort" only for the most complex agentic workflows.

Key Takeaways

The Open-Weight Gap: The lead of proprietary models over open-weight models has grown from 3 to 12 points .
Privacy as a Shield: 60% of the index is now private to stop benchmark contamination .
Agentic Shift: New benchmarks like FrontierCode 2.0 and Terminal-Bench-Science prioritize real-world action over static knowledge .
Cost Volatility: Per-task costs vary by more than 13x among top models, with GPT-6.1 Sol offering a high-efficiency alternative to Claude Opus 5.5 .
Specialization: No single model leads every test, meaning procurement must be based on specific task needs rather than aggregate scores .