Deep dive: The Third Scaling Law: Paying for Thinking per Task

Why today

Two releases this month were sold on how much thinking you can buy, more than on size. OpenAI's GPT-6.1 Sol comes with five reasoning settings, from low to max. It drops the option to switch reasoning off altogether. Google's Gemini 4 Argon also has adjustable reasoning, and it can now write responses up to 1 million tokens long. On Artificial Analysis, an independent benchmarking firm, Sol at max reasoning scored 52 on its Intelligence Index. That is one point below OpenAI's flagship, GPT-6 Astra. Sol did it for $0.72 per task against $3.26 for Astra. Argon tied Astra at 53. So the headline numbers are no longer just a model name. They are a model plus a reasoning setting plus a cost per task. Researchers call this test-time or inference-time compute scaling. It now drives both how good these products are and what they cost. Here is what it actually is.

Plainly

For most of the last decade, there were two ways to get a better model. Train a bigger one, or train it on more data. Both costs are paid once, up front. Inference-time scaling adds a third lever. You spend more compute while the model is answering. There are two basic ways to do it. The first is to let the model write a longer chain of reasoning before it answers: trying approaches, checking its work, backing up. The second is to sample many answers and pick the best one. Think of a junior analyst building a financial model. Give them five minutes and you get their first guess. Give them an afternoon and they try three structures, catch a sign error, and redo the assumptions. Same analyst, same training, better output, because they spent more effort on this one job. The catch is that you pay for that afternoon every time you assign the task. A 2024 paper from Stanford and Google researchers, titled Large Language Monkeys, made this concrete. They let an open coding model attempt real software bugs from SWE-bench Lite many times over. With one attempt it solved 15.9% of them. With 250 attempts, at least one attempt was correct 56% of the time. The previous best single-attempt result was 43%. There is an important qualifier. That 56% counts a problem as solved if any attempt was right. In practice you need a way to tell which attempt that was. In coding, unit tests can do that. In a legal memo or a strategy deck, nothing checks the answer automatically. The paper also found the gains were log-linear. Each tenfold increase in attempts bought a roughly similar step up. Every step costs ten times more than the last. Keep that shape in mind. It is the whole economic story. Spending on thinking works, but returns per dollar shrink fast, and the bill comes on every single request.

What the evidence shows

Some of this is well established. The Large Language Monkeys result held across tasks and models, and the log-linear pattern held over four orders of magnitude. A separate 2024 paper from Google DeepMind and Berkeley researchers, by Charlie Snell and colleagues, asked how to spend thinking compute wisely. Their key finding was that the best strategy depends on how hard the prompt is. Adapting effort to difficulty beat simple best-of-many sampling by more than 4x in efficiency. With compute held equal, a smaller model given extra thinking could beat a 14x larger model. That held on problems the smaller model could already sometimes solve, not on the hardest ones. Thinking helps a model use what it already knows. It does not supply knowledge the model lacks. This month's numbers come from Artificial Analysis, which is independent of the labs. It runs a composite of 10 benchmarks and reports cost per task. That figure uses token counts reported by each provider, combined with live cache hit rates. That matters, because a model that is cheap per token but thinks for longer can end up costing more per task. As Simon Willison put it, prices per token "aren't really comparable these days since what really matters is how many tokens a model takes to achieve a goal within an agentic harness." Now for what is claimed but not shown. OpenAI says Sol nearly matches Astra on agentic coding, computer use and professional work. That is self-reported. Artificial Analysis largely backs it, but only on its own benchmark suite. More thinking is also not always better. On Artificial Analysis' coding agent index, Sol scored 3 points higher at the xhigh setting than at max. Thinking costs time as well. Sol runs at 57.7 tokens per second, roughly two-thirds the speed of the previous Sol. Gemini's $1.99 per task is a launch discount. At standard prices it rises to $3.98, which is more than Astra. Argon's availability has also shifted. Google first limited it to its Fairwind cybersecurity program, and Willison reports it reached the Gemini API on October 13. And nobody outside these labs has published per-task costs from real customer workloads.

How people read it

Readers can interpret the same facts in different ways. The AGI-forward reading treats inference scaling as a whole new axis of progress, separate from pretraining. Toby Ord, an Oxford philosopher who studies AI risk, describes a loop. Models think hard to produce better training data, and that data is then distilled into faster models. If that loop compounds, progress could speed up even if pretraining stalls. The AI-realist reading focuses on the shape of the curve. Log-linear means each fixed improvement costs exponentially more. Ord estimates that 10x more reinforcement learning training buys about the same gain as 3x more inference. He argues that this pushes costs onto every query, where volume cannot spread them out. Realists also note that the gains are largest where answers can be checked, as in math and code. Gains are less clear in messy professional work. The market reading cuts against both. Epoch AI, a research group that tracks compute trends, argues the burden fades. In its words, the price to reach a given capability level falls very roughly a 5–10x cost reduction per year. Sol landing near Astra at about a fifth of the cost fits that trend. The policy reading is less discussed but important. Ord points out that rules keyed to training compute, like the EU AI Act and California's SB53, can miss a smaller model that thinks a great deal. The evidence on direction is fairly solid. How fast costs fall remains genuinely open.

What it means for builders and investors

Our read is that inference-time scaling turns a model's cost into a dial the product team controls. That changes how AI companies should be analysed. Cost of goods is no longer a fixed price per token. It is a spread that depends on task difficulty, reasoning setting and retries. Two companies on the same API can have very different gross margins. That depends on whether they send easy work to cheap settings and save max reasoning for the hard cases. That routing is real engineering, and it can be a modest moat. A wrapper that runs everything at max is exposed in two ways: competitors who route better, and providers who change prices. Argon's price doubling on December 1 shows how quickly the ground can shift. Latency matters too. Voice and real-time agents cannot always wait for a long chain of thought. That holds if the workload resembles the benchmark. One question worth asking in diligence might be: what is your cost per completed task, not per token, at each reasoning setting? And what share of your volume truly needs the top setting? We would want to see that measured on the company's own traffic.

What to watch

Three things would firm this up. First, Argon's standard pricing takes effect on December 1. We will see whether Google's cost per task holds up against Sol and Astra once the discount ends. Second, we'd want outside evidence beyond a single benchmark firm. Per-task costs from real agentic deployments, published by customers or independent testers, would show whether index gains carry over to production. Third, Epoch's trend implies the 52 Sol reaches today should cost a fraction of $0.72 within a year. If that happens, inference costs are a passing problem. If cost per task at the frontier keeps rising while scores climb only slowly, the realist reading gains ground. We should know by next autumn.