Tech • AI • Robotics
GPT-6 Astra: Do AI benchmarks no longer mean anything?
9.2/10TL;DR
The release of GPT6 Astra has reignited the debate over AI benchmarks: the same model can post 62.7% or 99.9% on ARC AGI 3 depending on the protocol used, showing that a raw score alone is no longer enough to judge performance.
KEY POINTS
Two official scores for the same model
On ARC AGI 3, GPT6 Astra reaches 62.7% with the standard Harness protocol, then 99.9% via a provider adapter. These two results do not contradict each other: they answer two different questions. The first measures what the model is worth in a common interface shared by all models; the second measures what it can do when it uses mechanisms designed specifically for it.
The interface strongly changes the result
The provider adapter mode notably makes it possible to preserve reasoning state across multiple interactions and to handle very long contexts better. In other words, the evaluation no longer measures only the model, but the model under specific technical conditions. A high score must therefore always be read together with its protocol, tools, and compute budget.
Even within one protocol, parameters matter
Tests on ARC AGI 3 compare several reasoning levels: no, low, medium, high, xi and max. Notably, the best result is not obtained at the maximum level: xi rises to 99.95%, versus 98.55% for max. This shows that a claim like “this model gets 99.9%” remains incomplete without the exact settings.
Several benchmarks are nearing the ceiling
Other published results show already very high scores: 97.6% on Frontier Math 4 V2, 96% on GPQA Diamond, 99.9% on ARC AGI 3, and 100% on Exploit Bench. This ceiling effect greatly reduces the comparative value of these tests. When several models converge toward 97 to 100%, the measurement tool no longer really distinguishes the best systems.
Contamination is becoming a central risk
The more public a benchmark is, the more likely it is to end up in papers, GitHub repositories, or datasets used for training. A 2024 study of 31 models in mathematical reasoning already reported substantial signs of contamination. This does not mean the models are only memorizing, but it does make score interpretation more delicate.
The case of Exploit Bench illustrates the problem
GPT6 Astra gets 100% on Exploit Bench, a cybersecurity benchmark. But OpenAI states that this score may be artificially inflated by the presence of historical vulnerabilities potentially known during training. On a new evaluation based on flaws disclosed between June and August 2026, and therefore after the model’s knowledge cutoff, the score drops to 39%.
The real issue is generalization
This shift from 100% to 39% completely changes how the result should be read. The first score may partly reflect a memory effect; the second tests more strongly the ability to generalize to unseen cases. The design of the benchmark therefore determines the question actually being asked of the model.
There is no “best AI” in general
Rankings vary by domain. GPT6 Astra clearly outperforms Claude Fable 5.1 on Frontier Math 4 V2, but falls behind on a university-level humanities exam. These reversals are a reminder that a benchmark measures a specific capability, not a single, stable general intelligence.
Toward living and more concrete benchmarks
New approaches seek to reduce these biases. LiveBench frequently refreshes its questions with recent sources to limit contamination. Other evaluations focus on open math problems or on the time horizon, meaning how long a task an agent can complete alone with a given probability—a measure closer to real usefulness and cost of use.
CONCLUSION
AI benchmarks remain essential, but reading them now requires the protocol, parameters, and test context. With GPT6 Astra, the gap between 62.7% and 99.9% shows that an isolated score no longer says enough about a model’s real capabilities.
Explain this