Grok 4.20's Fast Mode Needed 755 Turns to Get a Pokémon Red Starter on PokéBench

A Pokémon Red run that finished in mid-September became one of the most discussed agent benchmarks of the weekend after a post on r/LocalLLaMA on October 31 resurfaced it. On PokéBench, an open benchmark that gives every model the same 1,000-turn budget, xAI's Grok 4.20 in non-reasoning mode needed 755 turns just to receive a starter Pokémon, the second of ten milestones. The same model with reasoning switched on reached Viridian City, the fourth milestone, by turn 348.

That gap is the real finding. The viral framing centers on the model noticing its own loop and failing to escape it. The benchmark's own records point to something more useful for anyone deploying agents: on a long-horizon task, turning reasoning off did not save money. By Azure's own billing, the non-reasoning run cost $30.12 against $19.14 for the reasoning run that got more than twice as far.

According to the Reddit post, the model left Red's house by turn 36, then spent roughly 700 turns cycling through the opening area while the thought field it wrote each turn acknowledged the oscillation. The turn-36 exit and the quoted reasoning could not be independently verified; the 755-turn starter figure and the run's final standing come from the benchmark's public repository.

How PokéBench Scores an Agent Without a Human Judge

PokéBench is maintained by a developer publishing as VibeCodyH on GitHub, and it streams runs on Twitch with VODs on YouTube. Its premise is harness neutrality. Every model gets one screenshot, an ASCII walkability map, the on-screen text and up to six actions per turn. A turn is one model decision, so the score measures decision quality rather than GPU speed or token volume.

Scoring is read straight from game memory. The ten-rung ladder runs from leaving the house through getting a starter, reaching Route 1, Viridian City, Oak's Parcel, the Pokédex, Viridian Forest and Pewter City, up to beating Brock for the first badge. The furthest milestone reached wins, and fewer turns breaks ties. No human decides whether a run "counts."

The starter milestone is harder than it sounds for a vision-language agent. In Pokémon Red, the player cannot simply walk into Oak's Lab and pick a Pokémon. Professor Oak only appears when the player tries to leave Pallet Town through the tall grass at the north edge, then walks the player back to the lab. An agent that explores the town without trying that exit sees no new information, and every screen looks like progress that is not progress. The benchmark notes that Route 1 lands after the starter for exactly this reason.

Why an Agent Can Name Its Loop and Still Not Leave It

One harness choice explains why "the model noticed it was looping" is less surprising than it sounds. PokéBench deliberately stores history as facts, not narration: position, then actions, then new position and result. The maintainer added this "factual-history fix" because models kept re-reading their own wrong theories and recommitting to them. On the registry runner, history extends back to the second-most-recent milestone, trimmed to fit the configured context. For a run that has cleared only one milestone, that rule reaches back to the start of the game, so a model stalled before its starter is, in principle, looking at a long factual record of where it has already been. Each turn also carries a fresh screenshot and a walkability map, so the agent is not short of observations.

That design gives a stuck model plenty of evidence. A log showing the same few coordinates repeating makes the oscillation obvious to any reader, so a fast model can describe it in a sentence. What the log cannot supply is a new hypothesis about where to go. Detection is pattern-matching on recent context. Escape requires generating and testing an alternative plan — such as trying the town's north exit — the kind of deliberation that non-reasoning mode is built to skip.

Research on self-correction supports the distinction. Jie Huang and colleagues at Google DeepMind and the University of Illinois found that language models asked to review their own reasoning without external feedback often made accuracy worse, changing correct answers as readily as wrong ones. Noticing that something is wrong and knowing what to do instead are separate capabilities, and the second one benefits from extra inference.

xAI sells Grok 4.20 in reasoning, non-reasoning and multi-agent variants. The non-reasoning variant is positioned for speed and direct responses, skipping chain-of-thought in exchange for lower latency — a trade-off that worked against the model on a task requiring sustained replanning. PokéBench ran both single-model variants through Microsoft's Azure AI Foundry, which does not expose Grok's internal reasoning text, so viewers saw the model's stated plans but not the hidden deliberation behind the reasoning run.

The Grok Sweep Shows Reasoning Mode Doing the Real Work

The non-reasoning result was one of six Azure Grok runs the maintainer added to the leaderboard in a single sweep, all on the same prompt version and each with a recording verified against the run's wall-clock time.

Model and mode Furthest milestone Turn reached Wall time Grok 4.6 Beat Brock 316 10.3h Grok 4.1 Fast, reasoning Got the Pokédex 558 9.9h Grok 4.20, reasoning Reached Viridian City 348 11.1h Grok 4.3 Reached Viridian City 940 15.8h Grok 4.1 Fast, non-reasoning Reached Route 1 910 5.2h Grok 4.20, non-reasoning Got a starter 755 5.9h

The pattern holds across two generations. Both non-reasoning runs finished last among the Grok entries, and Grok 4.1 Fast in non-reasoning mode ended back in Red's house after being defeated near the end of its budget on Route 1. Grok 4.6 became only the second model to beat Brock on the board, behind Gemini 3.8 Flash at turn 279.

The cost figures need care. Rates were added to the registry after these runs, so the maintainer recomputed them from logged tokens. The two non-reasoning runs matched Azure's per-deployment bills to the cent. The reasoning runs did not, because the harness counts hidden reasoning as output while Azure billed only visible completion tokens, so the $19.14 Azure figure for Grok 4.20 reasoning is the billed amount and $31.45 is the harness estimate — a roughly $12 gap explained in PokéBench PR #43. Either way, the fast mode bought no savings: it spent about as much or more and went a fraction as far.

Pokémon Benchmarks Have Moved From Spectacle to Controlled Comparison

Pokémon became an AI test case in February 2025, when Anthropic streamed Claude 3.7 Sonnet playing Red and credited extended thinking with getting further than earlier models, which had wandered or looped. Claude then spent tens of hours trapped in Mt. Moon. Gemini 2.5 Pro later finished Pokémon Blue, but with a different harness, which its own developer warned made comparisons unreliable.

Academic work has tried to fix that. The PokéAgent Challenge, which ran as a NeurIPS 2025 competition, standardized infrastructure for comparing language-model, reinforcement-learning and hybrid agents and reported that Pokémon battling is nearly uncorrelated with 49 standard benchmarks. The PokéAgent paper is a preprint; its NeurIPS 2025 competition results have been independently confirmed. PokéBench is a smaller hobbyist effort, but it applies the same principle to the opening of Red: one fixed harness, public logs and the model as the only variable.

What One Run Can and Cannot Prove

PokéBench records a single seed per model, and the spec lists in-game randomness as a known source of variance. A second non-reasoning run might find Oak's trigger in fifty turns. The benchmark also requires reasoning models to run at recommended effort, so the non-reasoning entry is a deliberate contrast, not xAI's best configuration. And a Pokémon opening is not an enterprise workflow, even if the failure looks familiar to anyone who has watched a coding agent retry the same broken fix.

The quoted self-diagnosis also rests on the Reddit post alone. The run's full turn-by-turn log is committed to the repository, which means anyone can check how often the model described its loop and what it did next.

For teams choosing between a fast tier and a reasoning tier, the takeaway is concrete: per-token price misleads on tasks where an agent can stall. The number that matters is cost per milestone reached, and on this board the reasoning setting won on both progress and billed cost. The next useful data point is a multi-seed rerun of both Grok 4.20 modes on PokéBench's newer harness versions, which would show whether a 400-turn gap is a property of the model or a bad draw.