Harness-of-Harness: Shanghai AI Lab's Iterative Loop Boosts Coding Agents 52% Without Model Upgrade

When a coding agent writes a feature, breaks a different one while fixing the first, and then fails to notice because it is also the one checking its own work — that is not a model problem. It is a process problem. A new framework from Shanghai Artificial Intelligence Laboratory argues that the appropriate fix is not a larger language model but a different organizational structure around whichever model a team already runs.

The framework is called Harness-of-Harness, or HoH. Described in an arXiv preprint submitted September 1, 2026, it wraps an existing coding-agent harness in an outer planning-coding-testing loop, adds persistent state across iterations, and separates the roles that plan code, write code, and verify it into distinct agents — each with a deliberately limited view of the project. Across three benchmarks and three harness-model configurations, HoH averaged a 52.25% relative gain over the same agents run without the wrapper, with a maximum gain of 82.86%. In a five-day case study, the system built a playable first-person shooter game across more than 70 iterations without human intervention.

Why Coding Agents Stall on Long Projects

The submitted paper identifies four structural failure modes that specifically emerge when a coding agent works on a project over an extended period. Requirement amnesia refers to the loss of early architectural decisions and prior fixes as context compresses across a long session. Feature regression occurs when a change that appears locally valid silently breaks behavior elsewhere in the codebase. Development stagnation describes the pattern of repeatedly verifying already-confirmed components rather than advancing the implementation. And premature completion — arguably the most dangerous of the four — happens when the agent declares success without exposing missing functionality, delivering a structurally incomplete system.

These failure modes share a common root: in a single-session agent run, the agent that writes the code also evaluates its own work, the session's context is finite, and nothing carries verified behavioral evidence forward to the next pass. Adding more inference compute does not address the underlying architecture.

The framing matters because it reorients the engineering question. The dominant intuition in 2025 and early 2026 was that longer context windows or more capable models would eventually solve multi-file, multi-day engineering tasks. HoH's central evidence against that intuition is a direct token-budget comparison: a two-pass HoH run using 5.67 million tokens scored 64.84 on GameCraft-Bench, while three consecutive passes of the same vanilla harness using 6.33 million tokens scored only 58.24. The structural framework outperformed greater compute.

The Three-Role Loop: How HoH Works

Each HoH iteration runs three distinct agents in sequence, each using a separate harness-model invocation with strictly partitioned permissions.

The Project Planner receives the original software specification, the current artifact state, and the accumulated evidence from all prior QA evaluations. It produces a bounded development document for the next increment — selecting one locally completable objective that balances fixing outstanding issues with delivering one new capability. Critically, the Planner cannot write to the artifact; it is read-only. This prevents the Planner from anchoring on its own implementation assumptions when reading the current state.

The Developer receives the development document and the current artifact but not the prior evidence stream directly — a design choice that reduces anchoring on stale assumptions from previous test runs. The Developer is the only role with write access to the project, and it operates with shift-left testing: a sidecar testing tool runs concurrently during code writing, exposing regressions in real time rather than accumulating them for a later evaluation phase.

The QA Tester evaluates a frozen, read-only snapshot of the artifact. It cannot modify what it tests. Using both black-box evaluation (checking runtime behavior and rendered output) and white-box inspection (reading source code and runtime logs), it produces a structured evidence packet — documenting what works, what is broken, and what is missing. Claims not backed by test results are recorded as gaps rather than treated as successes. The evidence state produced by the QA Tester is one of the two quantities carried forward to the next iteration.

The other quantity carried forward is the artifact state itself — the code, configuration files, and assets at the end of the current iteration. Because the Developer warm-starts from the previous artifact, each pass builds directly on confirmed prior work rather than rebuilding context from a conversation history. The paper shows that removing warm-start decreases GameCraft-Bench scores by 7.85 points and increases token consumption by approximately 32% (from 8.41M to 11.12M tokens for three iterations with Codex plus GPT-5.5), because rebuilding the artifact from context costs significantly more inference than resuming from stored files.

All artifact changes are committed to Git with each iteration, providing a versioned checkpoint history. When a QA evaluation determines that the current iteration has regressed below the previous verified state, the framework rolls back to that checkpoint and incorporates the regression as a documented issue in the next planning cycle. Over the Fusepoint case study, 17 of 81 tracked issues were reopened due to regressions introduced by subsequent changes — each one surfaced by the independent QA Tester rather than discovered by the Developer reviewing its own output.

The paper formalizes five design principles derived from the architecture. Plans must balance fixing outstanding issues with adding one new capability — pure repair loops and pure feature additions each produce characteristic failure patterns. Changes must be small enough to confirm they do not break existing verified behavior. Implementation-time testing (the Developer's sidecar) must be separate from evaluation (the QA Tester's evidence packet). The framework constrains verifiable output formats rather than prescribing the agent's internal reasoning. And version history enables rollback when a candidate iteration degrades overall system quality.

Benchmarks: Three Tasks, Three Configurations

The HoH evaluation tested three harness-model pairs — Codex with GPT-5.5 (high), OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3 — across three benchmarks designed to cover different dimensions of end-to-end autonomous software development.

GameCraft-Bench tests whether a coding agent can build a complete, playable game from a natural-language specification inside the Godot engine. The benchmark contains 140 tasks across 15 game families, evaluated using replayed demonstrations and rubric-guided multimodal judging. It measures not just whether the agent generates syntactically valid code but whether the resulting artifact functions as an interactive game. The benchmark was released in June 2026; at publication time, the strongest single-pass agent scored approximately 41% on the overall metric, meaning most agents struggled to deliver games with coherent mechanics, visual feedback, and consistent presentation.

On GameCraft-Bench, the strongest HoH configuration — Codex plus GPT-5.5 (high) — rose from 49.58 (vanilla baseline) to 71.52 at HoH@3, a gain of 21.93 points. OpenCode plus DeepSeek-V4-Pro improved from 26.90 to 48.98 (+22.08 points), and Pi plus MiniMax-M3 rose from 42.16 to 58.78 (+16.62 points). Notably, weaker baselines gained proportionally more, suggesting that the HoH structure compensates partly for harness deficiencies.

FrontierSWE evaluates agents on ultra-long-horizon software engineering tasks — open-ended implementation, performance, and research problems given up to 20 hours per task. It uses a dominance metric: the fraction of head-to-head comparisons that a given configuration wins across multiple trials. On FrontierSWE, the gains were substantial across all three harness-model pairs: Codex rose from 44% dominance to 71% at HoH@3; OpenCode from 25% to 44%; Pi from 35% to 64%. The performance did not plateau at three iterations — running Codex to HoH@10 on FrontierSWE pushed dominance to 72.67%, compared with 27.33% for vanilla continuation. This result is among the paper's most significant findings: the iterative loop kept accumulating gains across ten cycles without saturating.

ProgramBench presents a cleanroom reconstruction task: agents receive only a compiled executable and usage documentation, then must build a codebase from scratch that reproduces the binary's behavior. It evaluates average test pass rate on hidden behavioral tests generated through agent-driven fuzzing. ProgramBench gains were positive across all configurations: +6.09 percentage points for Codex, +12.29 for OpenCode, and +16.85 for Pi.

The ablation study isolated each component's contribution. On GameCraft-Bench with Codex plus GPT-5.5, removing plan updates (fixing the plan after the initial creation) cost 8.13 points. Removing warm-start (forcing each iteration to rebuild the artifact from context) cost 7.85 points and increased token consumption. Removing evidence feedback (blocking the Planner from reading prior QA results) cost 6.28 points. No single mechanism produced all the gains; the framework requires all three working together.

The Fusepoint Case Study: Five Days, 70 Iterations, Zero Human Commits

To test HoH outside controlled benchmark conditions, the authors ran a five-day autonomous deployment using Codex CLI plus GPT-5.6-Sol (high) in Godot 4.7. The specification was a product requirements document describing player-perceivable content: a five-minute bomb-defusal mission, two sequential control-point objectives, eighteen enemy agents with AI behavior, and two narrative outcome branches (successful defusal and explosion). All engineering decisions — task ordering, architecture choices, and acceptance criteria — were determined by the framework.

The development trajectory divided into three phases. Iterations 1 through 27 established the executable skeleton; issues concentrated during this phase as the core systems were built. Iterations 28 through 49 ran feature development and regression repair in parallel, with the issue backlog fluctuating but not growing. Iterations 50 through 70 entered a stabilization phase in which active-issue count declined progressively as the QA Tester's evidence packets identified fewer new gaps and more existing ones closed.

Across 70-plus iterations, HoH tracked 81 unique issues, closed 65, left 16 open, and flagged 17 regressions — cases where a change that passed QA in one iteration broke previously verified behavior in a later one. Each regression was recorded as a documented issue rather than silently reverting undetected. The resulting game, Fusepoint, includes a rendered 3D environment, a complete weapon system, spatial audio, volumetric cloud and lighting effects, and the narrative branch structure specified in the original PRD. The project's full commit history, per-iteration development documents, issue lists, and QA evidence packets are publicly accessible in the repository. The team also built a pixel-art roguelite called Mournlight and a productivity application called Focusboard using the same framework.

What HoH Is Not Replacing

HoH does not modify the underlying coding-agent harness. Systems such as MetaGPT and ChatDev prescribe role workflows internally and require adopting the entire pipeline. Self-Harness, a related paper from the same period, evolves the harness itself — optimizing the scaffolding structure rather than organizing development around an existing one. HoH's architectural choice is deliberately conservative: it treats the harness as a fixed substrate, which means teams using Codex CLI, OpenCode, or Pi can adopt the outer loop without rewriting their agent configuration.

The gains also do not depend on frontier-model access. The Pi plus MiniMax-M3 configuration — the weakest of the three tested combinations — showed the largest proportional improvement on FrontierSWE (+29 dominance percentage points). This suggests the framework's value is not contingent on using the most capable available model; it addresses the process failure modes that affect any iterative coding agent.

What HoH does not yet answer is how the evidence-state accumulation scales beyond ten iterations. The paper tests T=10 only on FrontierSWE; for GameCraft-Bench and ProgramBench, T=3 is the maximum evaluated. As the evidence packet grows across dozens of iterations — as it did in the Fusepoint deployment — managing its structure, relevance weighting, and retrieval efficiency becomes a design question the current paper leaves open. The noted inconsistency between model names in the abstract and body (alphaxiv flags GPT-4o in the body, GPT-5.5 in the abstract) warrants checking against the final published version.

What Comes Next

Shanghai AI Lab has indicated plans to release HoH-lite, a lightweight implementation of the core workflow designed to lower the adoption barrier. The project page lists additional upcoming showcases. Whether the framework's gains hold for domain-specific or safety-critical software — where the QA Tester's black-box and white-box evaluation must be replaced with domain-specific test suites — is an open research question with direct implications for enterprise deployment.

The paper's deepest implication may be organizational rather than technical. For years, benchmark gains in coding agents were driven by model improvements: better reasoning, longer context, more training data. HoH's ablation evidence indicates that role separation, evidence persistence, and versioned rollback each contribute independently to multi-day autonomous development performance. The bottleneck, at least for this class of long-horizon task, appears to be the absence of a durable institutional memory between agent sessions — not the model's capabilities within any single one.