Tech • IA • Crypto
The Thinking Lever
6/10TL;DR
Increasing “test-time compute” by allowing AI models to use more tokens during reasoning significantly improves performance, but with diminishing returns and higher costs.
KEY POINTS
Test-Time Compute Boosts Performance
Advances in large language models show that allocating more computation during inference—known as test-time compute—improves problem-solving ability. As models spend more tokens reasoning through a task, accuracy rises across domains, including coding, scientific benchmarks, and complex question answering. Performance gains mirror those achieved by scaling model size during training.
Token Usage Directly Impacts Output Quality
Experiments with varying effort levels demonstrate clear trade-offs. Low-effort runs produce faster but simpler outputs, while higher-effort runs generate more detailed and accurate results. In one simulation task, increasing effort led to better vehicle behavior, more realistic environments, and improved adherence to real-world constraints, albeit with significantly higher token usage and latency.
Diminishing Returns at Maximum Effort
While higher effort improves outcomes, the gains taper off at extreme levels. Doubling or even multiplying token usage can yield only marginal improvements beyond a certain point. This creates a practical optimization problem between cost, speed, and intelligence, especially for production systems.
Three Core Components of Compute Usage
Test-time compute consists of three main elements: reasoning (thinking tokens), tool use, and text output. Reasoning acts as an internal scratchpad, tool use connects the model to external systems such as search or databases, and text forms the final response. Effective systems balance all three rather than disabling any component.
Adaptive Reasoning Improves Efficiency
New approaches such as adaptive thinking allow models to decide dynamically when to think, act, or respond. This replaces rigid pipelines where reasoning always precedes action. The adaptive model better reflects human workflows, alternating between thinking and doing, and can skip reasoning entirely for simple queries.
Effort Control Becomes a Key User Lever
Users can influence model behavior through effort levels and token budgets. Effort settings determine how long a model spends reasoning, while budgets constrain total compute. This introduces a trade-off: higher intelligence versus lower latency and cost. Choosing the right balance depends heavily on the task.
Guidelines for Effort Levels
Medium and low effort suit tasks like classification, summarization, or extraction where speed matters more than deep reasoning. High and extra-high effort provide stronger performance for complex tasks. Maximum effort is best reserved for the most difficult problems due to cost and diminishing returns.
Model Size Still Matters More Than Effort
Larger models consistently outperform smaller ones, even when smaller models use higher effort levels. Comparisons show that increasing effort on a small model cannot fully compensate for lower baseline capability. For tasks requiring intelligence, using a more capable model at moderate effort is often more effective.
Unexpected Behaviors at Low Effort
Constraining reasoning can produce unconventional strategies. In a gaming benchmark, a low-effort model adopted shortcut-driven tactics to progress faster, avoiding unnecessary actions. This suggests that limiting compute can sometimes lead to creative, if less thorough, solutions.
Toward Autonomous Compute Allocation
The long-term goal is for models to autonomously allocate compute based on task complexity within predefined budgets. This would allow systems to scale from seconds of work to hours or longer, handling increasingly complex tasks while respecting cost and time constraints.
CONCLUSION
Test-time compute has emerged as a critical lever for improving AI performance, but optimal use requires balancing cost, latency, and diminishing returns while pairing effort levels with appropriate model size.