Paused-Game Benchmarks Overrate AI Agents: RealtimeBench Exposes Real-Time Performance Gap

Every major AI agent benchmark today makes the same assumption: the game pauses while the model thinks. A paper listed on arXiv this week demonstrates that assumption is distorting the picture. When researchers let games keep running during inference, the performance rankings of leading models reversed — and a new training method called TORSI managed to nearly double a fast agent's real-time score without requiring any model retraining.

The paper, "The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock" (arXiv:2610.20665, an unreviewed preprint), introduces RealtimeBench and argues that paused-game evaluation has been systematically inflating agent capability since the first game-playing benchmarks were deployed.

RealtimeBench Introduces the Clock That Standard Evaluations Ignore

RealtimeBench spans 180 tasks across six game types, evaluating agents under three distinct conditions: the standard paused-clock setting used by prior benchmarks, a live-clock setting where the game continues running while the model reasons, and an accelerated-clock condition that runs the environment faster than real time. The paused condition replicates what essentially every current agent leaderboard measures. The live and accelerated conditions add the temporal cost that deployed agents actually face.

The design exposes a fundamental mismatch between benchmarked capability and production performance. In a real deployment — whether an agent managing a trading system, navigating a multiplayer game, routing customer-service requests, or operating any environment that doesn't wait for deliberation to finish — actions arriving late are actions arriving wrong. The agent's score under live-clock conditions is not merely a variant of its paused score; it reflects an entirely different performance dimension.

The specific six game types used in RealtimeBench are not detailed in the paper's abstract. Those details and the per-game score breakdowns are available in the full paper.

GPT-6.1-sol Leads on Paused Games, Then Falls Behind When the Clock Runs

The most striking empirical finding in the paper involves two models that represent genuinely different architectural paradigms. GPT-6.1-sol, OpenAI's September 2026 reasoning model positioned as a near-top-tier option at roughly one-fifth the cost of GPT-6 Astra, outperforms Jev when game time is paused during inference. When the clock runs at normal or accelerated speed, the ranking reverses.

This reversal is not a subtle statistical effect. It demonstrates that GPT-6.1-sol's advantage in the paused setting comes from the time it spends reasoning — time the benchmark was granting it for free. Under live-clock conditions, that deliberation time becomes a cost, and Jev's architecture makes it far better equipped to absorb that cost.

TypeSafe AI describes Jev as a "System One Model," after the fast, intuitive reasoning in Daniel Kahneman's two-system framework. Jev does not generate text. Developers define a finite set of possible outputs — a choice among options, a yes/no probability, a numeric score — and Jev evaluates and returns those in parallel, each with a calibrated confidence value, in 70 to 500 milliseconds. TypeSafe, founded in 2024 by Diogo Almeida — a co-inventor of the RLHF technique behind ChatGPT — exited stealth on September 15, 2026 with $40 million in seed funding led by DCVC.

The architectural difference between Jev and GPT-6.1-sol is not just latency — it is the underlying task format. GPT-6.1-sol produces chains of reasoning tokens that take seconds; Jev evaluates a predefined schema in milliseconds. In a paused benchmark, GPT-6.1-sol's reasoning is an asset. In a live environment, it is a liability. RealtimeBench is, in effect, the first framework to penalize that liability.

Read more: TypeSafe's Jev Delivers Structured AI Decisions Without Generating a Single Word

TORSI Teaches Agents to Think Faster by Separating Deliberation From Action

Two-Timescale Offline Recursive Self-Improvement addresses the real-time performance gap by explicitly separating the timescales of learning and action. The method has two components that operate at fundamentally different speeds.

The slow timescale is an offline teacher. After a round of gameplay, the teacher examines recorded failures in a paused-clock setting — where it has unlimited time to analyze what went wrong. It tests candidate corrections, and writes a set of reusable natural-language instructions summarizing what the agent should do differently. A validation step gates which instruction updates are actually adopted: candidate instructions must improve performance on held-out validation games before they become part of the student's playbook.

The fast timescale is the student, which acts during live gameplay using the teacher-written instructions, but without calling the teacher during inference. The student does not need to perform the teacher's slow analysis at runtime; it acts from the distilled guidance the teacher has already prepared. This is a practical formalization of how training and deployment are actually separated in real systems, turned into a repeatable improvement loop.

According to the paper's reported results, Jev's average test score across all six game types rose from 13.51 to 24.19 on a 0-to-100 progress scale over two TORSI rounds — a gain of just over 10 points and roughly 79 percent relative improvement.

The transfer finding is the most immediately useful result for practitioners: the same natural-language instructions that improved Jev's scores also improved average scores for three other model families when applied without retuning. The instruction set captures game-relevant behavioral knowledge in a form that is not architecture-specific.

A Score of 24 Out of 100 Puts the Advance in Context

The improvement is genuine, but the absolute numbers matter. Even after two TORSI rounds, Jev's average test score of 24.19 out of 100 indicates that substantial performance headroom remains. The benchmark is calibrated to real-time difficulty, which is intrinsically harder than paused evaluation, but a 24 percent progress score means the tasks are still largely unsolved.

The paper runs two rounds of TORSI and demonstrates monotonic improvement with each round. Whether additional rounds continue to yield gains, at what rate improvement diminishes, and whether the system can eventually saturate these tasks are open questions the paper does not answer. The recursion in "Recursive Self-Improvement" depends on a continuing supply of novel failures for the teacher to analyze; whether failure modes become repetitive after several rounds is an important constraint not yet characterized.

It is also worth noting that the specific game types in RealtimeBench are designed environments with discrete action spaces and defined win conditions. Real-world deployment domains — robotics navigation, live financial decision-making, autonomous system monitoring — introduce continuous action spaces, noisier state representations, and latency profiles that may not generalize directly from game results.

Jev itself has not been independently benchmarked by researchers outside TypeSafe AI; the company's claims of up to 444 times lower cost and 193 times faster inference compared with larger LLMs are company-reported figures that TypeSafe explicitly characterizes as upper-bound estimates on workflows optimized in-house.

Temporal Cost Is the Variable That Agent Benchmarks Have Omitted

The broader implication of RealtimeBench is that the current benchmark ecosystem — SWE-bench, WebArena, AgentBench, tau-bench, and the dozens of task-specific leaderboards that have proliferated since 2024 — are collectively measuring agent capability on a dimension that partially misrepresents production performance. When the environment never waits for the model, inference latency is not just a cost to optimize away; it is a constraint that changes which models are appropriate for which tasks.

Read more: New Benchmark Shows Top AI Models Score Below 20 When Building Agent Systems From Scratch

This does not mean slower, more capable reasoning models are less valuable — GPT-6.1-sol and models like it remain better suited for tasks where the environment is amenable to deliberation, such as offline document analysis, code review, or research summarization. But for the growing class of deployments where state is continuously changing — agent-mediated gaming, live data pipelines, real-time monitoring systems, interactive robotics — a ranking established on paused benchmarks offers a misleading guide.

If RealtimeBench's design principle is adopted more broadly, the consequence is a clearer segmentation of the agent model market: real-time decision models like Jev compete in one regime, deliberative reasoning models compete in another, and the relevant benchmark for a deployment is the one that matches the deployment's temporal structure. The next methodological step would be extending live-clock evaluation beyond game domains — a challenge the paper's authors leave for future work.