Google DeepMind Introduces Dream-RSI to Let Coding Agents Improve Their Own Search Strategy

Researchers from Google DeepMind, the University of Maryland College Park, and the University of Virginia have published a framework called Dream-RSI that enables coding agents to recursively improve how they search for solutions — without modifying the underlying code-generating model or its evaluator. The paper, posted on arXiv on September 14, 2026, addresses one of the central bottlenecks in long-horizon AI-driven discovery: the exploration strategy itself has historically been fixed by hand and never updated from experience.

Dream-RSI's core idea is that a completed discovery run already contains everything needed to evaluate alternative search strategies. Each run produces a structured tree recording which workspace a coding agent continued from, what it generated, and what score the evaluator returned. Rather than re-running the agent and evaluator from scratch to test a new policy, Dream-RSI replays candidate policies against that recorded tree at zero additional execution cost. A policy-development agent then revises the exploration code based on those replay results, and the improved policy is redeployed for the next real search round, which in turn expands the tree pool for further refinement.

The team tested Dream-RSI across eight tasks in three domains — algorithm engineering, mathematical optimization, and GPU kernel engineering — using Google's Gemini-3.1 Pro and Gemini-3.7-Flash models as the coding agents. Against a controlled baseline that kept the exploration policy fixed, Dream-RSI produced consistent improvements in either the number of agent calls required or the quality of the programs found.

👉 Read more: GPTS24's coverage of recursive self-improvement and the verification bottleneck in AI systems

Why the Search Controller Had Never Been Improved Before

When a coding agent hunts for a faster Lasso solver or a more efficient GPU kernel, every candidate program it writes costs an actual generation-and-evaluation cycle. That expense is unavoidable: you cannot know whether a program is faster without running it. But the search controller — the layer that decides which workspace to continue from, how many parallel attempts to run at once, and when to open a new branch instead of refining an existing one — does not generate or execute code. It orchestrates the agent that does.

The problem is that improving the controller is almost as expensive as running the discovery loop itself. To know whether a new search policy is better, you must let it steer many generation-and-evaluation cycles until a meaningful trajectory has accumulated. If you update the policy every time and re-run the full loop from scratch, policy development quickly consumes the budget that was supposed to be spent finding good programs.

Dream-RSI solves this by separating the two costs. Real discovery runs proceed normally, building trees of attempts with their evaluation outcomes stored at every node. Once a tree exists, a new candidate policy can be tested against it by traversing its branches in a different order — prioritizing different nodes, batching parallel attempts differently, stopping at a different point. Because every node's outcome is already recorded, this replay is instant. Thousands of candidate policies can be evaluated against the same historical tree for the cost of reading a file, not running an agent.

The authors draw an analogy to model-based reinforcement learning: just as a world model allows a reinforcement learning agent to "dream" about the consequences of actions without physically taking them, the accumulated discovery tree allows the exploration policy to dream about how different strategies would have played out — and then be updated accordingly before the next real deployment.

How the Replay Mechanism Works

A discovery tree is rooted at an initial workspace. When the exploration policy selects a node to continue, the coding agent picks up that node's saved file state and prior observations, generates a new candidate program, and the evaluator scores it. The result — code, diagnostics, and score — is stored as a child of the selected node. The policy can also return to the root at any point to open a new branch rather than extending an existing one.

In offline replay, the same decision interface applies. A candidate policy begins with only the root node visible. Each time it selects a node, the historical tree reveals what was actually recorded there — the child the real agent generated and the evaluation it received. This continues until the policy selects an empty batch, reaches a round limit, or the entire recorded tree has been revealed. Critically, the policy never sees anything that was not historically observed in the order it was observed; there is no cheating from future knowledge.

The replay objective weighs three factors simultaneously: the best program score found among revealed nodes, the number of generation-evaluation requests the trajectory used (penalizing waste), and a bonus for useful parallelism — rewarding policies that batch independent continuations rather than executing sequentially. A fixed policy-development agent examines replay trajectories and scores across versions, then rewrites the exploration policy code. The selected version must score at least as well as the current policy on the accumulated history, providing a non-worsening guarantee in replay.

After each offline phase, the updated policy goes back online. The real run it produces extends the historical tree pool, giving the next offline phase more worlds to dream against — and allowing the policy to encounter search conditions that no earlier strategy would have reached.

Read more: Study finds context overflow, not compression quality, is the primary driver of coding agent failure

Lasso Results: Fewer Calls, Better Solvers

The most detailed results in the paper concern the Lasso regularization path problem: given a feature matrix and a sequence of regularization strengths, find an implementation that computes the complete solution path faster than standard libraries while remaining numerically correct. This benchmark, used by the SimpleTES system, requires discovering a compiled program, not a Python wrapper, and verifies correctness on independent instances before measuring wall-clock runtime.

Both methods started from the same hand-written parallel exploration policy in round one. In subsequent rounds, the fixed-exploration baseline kept that policy unchanged; Dream-RSI revised and redeployed it after each round.

With Gemini-3.1 Pro, the fixed baseline used 550 cumulative agent calls across five rounds and produced a solver that averaged 3,587.1 milliseconds across six held-out downstream datasets. Dream-RSI used 317 calls and achieved an average of 2,931.0 milliseconds — 18 percent faster at 42 percent lower discovery cost. With Gemini-3.7-Flash, the fixed baseline used 3,200 calls for an average of 2,516.7 milliseconds; Dream-RSI used 1,879 calls for 2,350.6 milliseconds.

The two Gemini variants found qualitatively different programs. The Pro model produced a solver with particular strength on RCV1, the largest dataset in the benchmark, cutting its runtime from 19,550.1 to 14,616.0 milliseconds — a reduction of more than 25 percent on that dataset alone, though it was slower than the fixed-exploration solver on the five smaller datasets. The Flash model produced a more general-purpose solver that improved on five of six datasets and was slower on one.

The paper also compares against SimpleTES, which uses a GPT-OSS-120B model with 51,200 reported generations. Dream-RSI's Pro configuration uses approximately 162 times fewer discovery-agent calls while reaching lower average downstream runtime on the same six held-out datasets — though the two systems use different models and search implementations, so this comparison reflects relative scale rather than a controlled head-to-head.

Mathematical Optimization: Competitive at a Fraction of the Cost

The paper tests three mathematical optimization tasks over ten rounds with Gemini-3.1 Pro: the Sum-Difference problem (finding an integer set whose sumset grows faster than its difference set), Circle Packing (fitting as many circles as possible in a unit square to maximize total radius), and Autocorrelation Inequalities (minimizing the peak of a function's autoconvolution).

Dream-RSI reached a Sum-Difference score of 1.145427, ahead of SimpleTES at 1.143975 and the fixed-exploration baseline at 1.144047. On Circle Packing it matched the strongest reported result across all compared methods at 2.635983. On the Autocorrelation task, it obtained 1.456375, slightly behind the fixed-exploration baseline at 1.456001 and behind SimpleTES's leading 1.453675 — though SimpleTES required 51,200 generations compared with fewer than 1,000 for Dream-RSI's approach.

The results show that the exploration controller generalizes across problem types, but does not guarantee improvement on every objective. The Autocorrelation result illustrates that the replay-based policy can fail to find a better exploration strategy when the search landscape does not reward the kinds of branching and parallel decisions the policy learns to make.

GPU Kernel Engineering: Efficiency vs. Peak Performance

The four kernel-engineering tasks from the KernelBench benchmark — VGG16, LayerNorm, ConvDiv, and ConvMax — test different performance dimensions of Dream-RSI.

On VGG16 and LayerNorm, both methods ultimately reached comparable program performance, but Dream-RSI got there with 2.43 times and 1.79 times fewer generated candidates respectively. On ConvDiv and ConvMax, both methods were given a similar generation budget, and Dream-RSI's programs scored 2.09 times and 1.44 times higher in execution speed (measured as the inverse of milliseconds per operation).

The paper notes that kernel engineering requires the agent to reason jointly about algorithmic structure, memory access patterns, parallelization, and hardware-specific constraints — a substantially different problem from the Lasso path or discrete mathematical optimization. That Dream-RSI's exploration improvement transferred across all three domains suggests the replay mechanism is not narrowly tuned to one type of search space.

Prompt-Level Guidance Consistently Hurts

One ablation in the paper addresses a natural alternative: instead of using historical trees as a structured replay simulator, summarize the prior search trajectory into high-level directional insights and inject them into the next round's prompt. The authors tested this on the ConvDiv kernel task, applying such prompt-level guidance to both the fixed-exploration baseline and Dream-RSI.

In every case, adding the guidance made performance worse. The authors interpret this as evidence that, in long-horizon discovery with many parallel threads, strong semantic biases about where to search tend to over-constrain the space and suppress the diversity that makes parallel exploration valuable. Using history as a replay simulator — to refine how the controller allocates attempts — produces different and more durable improvements than telling the agent what directions to prefer.

An Adaptive Search Budget

In additional analysis on the ConvDiv task, the paper documents how the exploration policy's behavior changed across nine rounds. The number of evaluated attempts per round dropped from 110 in the first round to around 50 as performance improved rapidly — the policy learned to concentrate effort. When progress plateaued, the attempt count rose again, reaching roughly 90 per round, and was followed by further performance gains. This adaptivity emerged from the policy-improvement loop rather than being prescribed in advance.

The Dream-RSI project page and code repository are publicly available. The paper describes the framework as directly applicable to any agent-driven discovery setting where outcomes can be evaluated automatically, with the quality of the evaluator remaining the binding constraint on what the self-improvement loop can achieve.