A Thirst for Reality: Unpacking GPT-6 Astra

The Transformer architecture that every modern language model is built on was invented at Google in 2017. OpenAI bet on scaling that architecture when few people believed in it, and has shipped a flagship model a year ever since. The question is where exactly the progress happens in each new release.

On 3 September 2026 OpenAI released GPT-6 Astra and announced the arrival of the AGI era. Company president Greg Brockman summed the release up in a single phrase: the ARC-AGI-3 benchmark is fully solved.

A few hours later, the organization that actually builds that benchmark published its own measurement. It differed from the OpenAI number by almost forty percentage points.

What follows are four numbers that say more about this release than all the marketing around it.

ARC-AGI-3 tests the ability to figure out an unfamiliar interactive environment. The model lands in something that was not in its training data and has to work out the rules as it goes. That skill is usually called the line between a human and a statistical model. OpenAI reported 99.9%. ARC Prize, the organization behind the test, measured 62.71%. Both numbers are real, and the difference lies in how the model was run.

This is where the harness matters: it is not the neural network itself, it is the scaffolding, the code around the model, an external program that runs the model, stores its notes, feeds the task in chunks and collects the answer. OpenAI used its own harness: it preserved the model's internal reasoning between steps and compressed a long dialogue intelligently. ARC Prize uses a standard one, identical for every model.

The ARC Prize verdict: the gap is caused by memory handling, but the neural network is the same in both cases. Thirty-seven percentage points of the gain came from the code around the model, which is to say from the harness.

ExploitBench checks whether a model can find vulnerabilities in software. Astra scored 100%.

The same test, rebuilt without historical vulnerabilities, meaning without the holes already described on the internet and swept into the training data, gave 39%.

This is exactly the argument that has been running for years, shown as a number. Where the task has already been solved before, the result is close to perfect. Where the task is genuinely new, it drops by a factor of two and a half.

Yes, 39% on finding unknown vulnerabilities is still a lot. OpenAI had good reason to assign the model a critical cyber capability level and to hold part of its functionality back from public access. But the difference between 100% and 39% mostly speaks to the system having a good memory for the already known. It says far less about the ability to find something fundamentally new.

Artificial Analysis, an independent model comparison platform, gave Astra 61.2 points on its overall intelligence index. It gave the previous model, GPT-5.6 Sol, 60.9. Three tenths of a point per generation. On agentic coding, meaning tasks where the model works on its own for a long stretch: writes code, runs it, sees the error, fixes it, there is a gap as well: 67.0 versus 65.1. There is progress, and it is once again where the model works in a loop with external tools.

According to the official pricing, Astra costs 10 dollars per million input tokens and 50 per million output tokens. That is two and a half times more expensive than the previous flagship.

So: a near-zero gain on the overall intelligence index at a price that went up two and a half times.

The gain shows up where the scaffolding got better: memory between steps, working a computer, long chains of actions. Where the model is measured on its own, the gain sits inside the margin of error, and the price has multiplied.

Progress has moved out of the neural network and into the engineering around it.

Into memory management, into sandboxes for running code, into the loops of tried it, saw the error, fixed it, into protocols for working with external tools.

This is a normal phase for any technology. The jet engine was invented once, and the next forty years went into the airframe, the fuel, the instruments and the maintenance.

François Chollet, the author of ARC-AGI and one of the harshest critics of overrating language models, moved his AGI forecast closer after Astra. His phrasing: progress is running roughly twice as fast as he expected. He separately noted that on efficiency of action Astra matched a human for the first time. The average human scores about 48% on this test, Astra 62.71%. Epoch AI put Astra first in its capability index with a record 169 points. In other words, the people who professionally catch models memorizing are saying that something fundamental happened.

The jump from 7.8% for GPT-5.6 Sol and 30.2% for Claude Opus 5 to 62.71% is a fact, the progress is real. But the context matters: Chollet himself stressed that the test results do not prove AGI has arrived, and pointed to the difference in harnesses as the reason for the gap in the numbers, calling a direct comparison of 99.9% and 62.7% incorrect. A situation where the author of a benchmark has to clarify how exactly an AI giant is using its results is rare and telling.

The gap between what models can already do and what is actually wired into business processes is currently much wider than the gap between model generations. While the market argues about whether the AGI era has begun, the real unfinished work sits two levels down: integrations, memory, access, error handling, human verification of the result.

The gain OpenAI charges two and a half times more for is largely something you can build yourself. Most of it is concentrated in the scaffolding around the model, and ordinary AI engineers are already writing that scaffolding.

Good news for the people building systems. Bad news for the people waiting for the next model to solve everything for them.