HarnessPro Cuts AI Agent Costs in Half While Matching State-of-the-Art Pass Rates

A research team from Tsinghua University and Zhipu AI released a new agent-optimization framework last week that flips the prevailing assumption in recursive self-improvement: that making AI agents better at tasks inevitably makes them more expensive to run. HarnessPro, published October 19 on arXiv and open-sourced simultaneously on GitHub, shows that modifying the scaffolding around a foundation model — rather than retraining the model itself — can improve both pass rates and per-task cost at the same time, as long as cost is treated as an optimization target rather than an afterthought.

On SWE-Bench Pro, Scale AI's 1,865-problem repository-level coding benchmark, HarnessPro's medium-performance configuration matched the previous state-of-the-art label-free method (Microsoft's RHO) on pass rate while cutting average per-task spend from $2.68 to $1.26 — a 53% reduction. Its high-performance configuration achieved the highest published pass rate on Terminal-Bench 2, an 89-task CLI benchmark from an ICLR 2026 paper, while simultaneously spending 33.9% less per task than RHO. The same RHO method had increased per-task cost by 42.6% on SWE-Bench Pro relative to the unoptimized baseline while improving pass rates — an exchange that makes deployment economics unworkable for many production use cases.

What makes HarnessPro structurally different from its predecessors is not any single technique but a two-part architectural decision: it treats cost and pass rate as joint objectives, selecting harness configurations from the Pareto frontier rather than optimizing for one metric, and it replaces the LLM-as-judge evaluation method (which the paper shows has 41.7% recall on failures) with synthesized executable unit tests. Both choices change what the system optimizes toward. The result is not one evolved harness but three — Lite, Medium, and Heavy — produced by a single evolution run and targeting different points on the cost-performance trade-off.

The corresponding authors are Jie Tang, a professor in Tsinghua's computer science department and chief scientist at Zhipu AI, and H. Vicky Zhao, an associate professor in Tsinghua's automation department. Co-first authors Zhuoshi Pan and ZhiYuan Feng are both affiliated with Tsinghua.

The RSI Cost-Inflation Problem the Rest of the Field Has Not Solved

To understand what HarnessPro is responding to, it helps to understand the specific failure mode it identifies in prior work.

Recursive self-improvement of agent harnesses works like this: a foundation model runs on a set of tasks using some initial scaffolding configuration — prompts, tool definitions, retry logic, memory management, and orchestration code — and the resulting execution trajectories are fed back to another model (or the same one) that proposes improvements to the scaffolding. The improved scaffolding runs again, produces new trajectories, and the loop repeats. No human labeling of correct outputs is required; the system grades itself.

The problem, as Tsinghua's team frames it, is that the most straightforward way to improve pass rate is through more thorough exploration and more extensive verification. More exploration means more tool calls, more reasoning steps, and more tokens. More verification means running additional checks, querying models more often, and increasing the number of inference passes before committing to an answer. Both increase cost.

Microsoft Research's RHO framework, published in June 2026, demonstrated this pattern clearly. Using a label-free retrospective evaluation method in which a language model judges whether each trajectory succeeded, RHO achieved meaningful pass-rate improvements across coding benchmarks. But on SWE-Bench Pro, it did so while increasing the average per-task cost from $1.88 to $2.68 — a 42.6% increase. The trajectory toward higher-scoring harnesses led, via single-objective optimization, toward more expensive harnesses.

This dynamic is not unique to RHO. It reflects a structural feature of single-objective RSI: when the only signal is task success, the system has no incentive to find cheaper paths. Harnesses evolve to do whatever maximizes the pass metric, regardless of token cost. In a research benchmark context, this is acceptable. In a production deployment context, where each task runs against real API billing, a 42% cost increase per RSI-optimization pass makes the technology difficult to justify.

HarnessPro's premise is that addressing this requires treating cost as a first-class citizen of the optimization objective from the beginning — not as a regularization term added later, but as a co-equal axis in candidate selection.

A Four-Agent Evolution Loop That Diagnoses, Plans, Edits, and Tests

HarnessPro's evolution process runs one epoch at a time, with each epoch processing the entire pool of candidate harnesses through a four-stage pipeline before the next selection round.

The first stage is a Trajectory Analyzer. Rather than evaluating only whether a harness succeeded or failed on each task, the Trajectory Analyzer reads three independent execution rollouts for each candidate harness on the training set. Three rollouts reduce noise from the stochastic behavior of language models under sampling; single-pass evaluation would conflate genuine harness failure with unlucky draws from the model's distribution. From these three rollouts, the Trajectory Analyzer identifies not just outcomes but patterns: which tool-call sequences were redundant, where the agent got stuck in retry loops, where context was being populated with information that never influenced the final output. One case study in the paper's appendix illustrates the scale of what this catches. In a task involving mail server configuration, the agent successfully completed the setup — but the Trajectory Analyzer flagged that it had repeatedly dumped irrelevant configuration-file logs during intermediate steps, consuming 2.51 million input tokens on material that contributed nothing to the final answer. No pass/fail metric would have surfaced this; only trajectory-level analysis could.

The second stage is an Evolution Planner. It receives the Trajectory Analyzer's diagnosis along with cost metrics (tokens consumed, API spend) and the current code diffs between candidate harnesses in the pool. The diff comparison is deliberate: it allows the Evolution Planner to observe which modifications other candidates have made and assess whether successful adaptations can be borrowed. For each candidate harness, the Evolution Planner produces three independent modification proposals: one weighted toward cost reduction (I₁), one seeking a balanced trade-off (I₂), and one pursuing maximum performance without cost constraint (I₃). This three-branch design means that even a harness that was initially optimized for cost will generate a performance-focused variant, and vice versa — preventing premature convergence on a single region of the trade-off space.

The third stage is a Harness Editor. It takes each Evolution Planner proposal and executes it in an isolated workspace: modifying the harness's Python code, configuration files, and prompt templates. Every modification is committed to Git, providing a complete version history that enables rollback if a candidate's performance degrades unexpectedly. The use of an isolated workspace means that edits from one harness lineage cannot contaminate another; candidates in the pool evolve independently until selection occurs.

The fourth stage — and the one with the most methodological significance — is the Test Synthesizer. In a label-free setting, evaluating whether a harness's output is correct requires some form of automated judgment without access to reference answers. The standard approach is LLM-as-judge: present the execution trajectory to a language model and ask it to determine whether the task was completed correctly. The paper reports that this approach has a 41.7% recall rate on failures — meaning that when a harness actually failed a task, the LLM judge identified the failure as a failure only 41.7% of the time. The other 58.3% of failures were rated as successes. This is a form of systematic optimism bias well-documented in the LLM evaluation literature, particularly pronounced in tasks where success requires precise terminal state verification rather than qualitative text judgment.

The Test Synthesizer replaces LLM judgment with generated code. Given a task description, the Test Synthesizer writes ten Python unit tests using Python's unittest framework. These tests verify specific, deterministic properties of the output — not general impressions. A test either passes or fails; there is no probability distribution over success. The resulting recall rate on failures rises to 83.3%, a 41.6-percentage-point improvement over LLM judging. Because the tests are generated from task descriptions rather than reference solutions, no labeled data is required — the label-free property is preserved.

What the Benchmark Numbers Show and What They Do Not Prove

After each epoch of evolution, all candidate harnesses are evaluated against a test set, and their positions on the cost-vs.-pass-rate plane are determined. Candidates that are Pareto-dominated — meaning another candidate exists that is both cheaper and more accurate — are eliminated from the pool. From the surviving non-dominated candidates that form the Pareto frontier, HarnessPro selects three specific harnesses: one closest to the minimum-cost objective (Lite), one closest to a balanced target (Medium), and one closest to the maximum-performance objective (Heavy). The selection uses normalized weighted squared distance to these targets, with different weight vectors emphasizing cost or performance as appropriate to each tier. All three selected harnesses enter the next evolution epoch as a starting pool, allowing each tier to continue improving within its cost-performance region.

The paper evaluates these three configurations across three benchmarks using two model pairings. The GPT-5.5 + Codex combination and the GLM-5.2 + Claude Code combination test whether evolved harnesses generalize across model families. All performance figures that follow are self-reported in the paper; no independent reproduction by external researchers has occurred at the time of this writing, five days after the arXiv submission.

On Terminal-Bench 2, which tests 89 command-line interface tasks, HarnessPro Heavy achieved an 80.8% average pass rate — 4.5 percentage points higher than RHO and representing a claimed state of the art. Terminal-Bench 2 set its frontier-model baseline ceiling below 65% when the benchmark was published at ICLR 2026, making the 80.8% figure significant if reproduced independently. Per-task cost under Heavy was 33.9% lower than RHO.

On SWE-Bench Pro, which requires resolving real GitHub issues across 1,865 repository-level problems, HarnessPro Medium achieved 79.0% — approximately matching RHO's 80.0% — at $1.26 per task versus RHO's $2.68. HarnessPro Heavy exceeded RHO's pass rate while spending less. The SWE-Bench Pro results come with a benchmark-level caveat: the paper's authors note that contamination remains a concern, as the models underlying the harness — GPT-5.5 or GLM-5.2 — may have encountered related repository patterns during pretraining. HarnessPro modifies harness code rather than model weights, which reduces but does not eliminate this concern.

On ProgramBench, which requires program reconstruction from scratch, all three HarnessPro tiers outperformed both the unoptimized baseline and RHO across both model pairings.

HarnessPro Lite — the minimum-cost tier — achieved pass rates above the unoptimized vanilla baseline on all three benchmarks while cutting spend by 17.7% to 57.4% versus vanilla. This is an unusual result: a configuration optimized primarily for cost nonetheless improves over the unoptimized baseline, suggesting the evolution process found genuine efficiency improvements rather than simply cutting agent capability.

A secondary finding that was not a design objective: on SWE-Bench Pro Heavy, average task completion time fell from 520 seconds to 412 seconds — a 20.7% reduction in latency, despite latency not being included in the optimization objective. The authors attribute this to the same token-efficiency mechanisms that reduce cost: when the harness wastes fewer tokens on redundant operations, it also completes tasks faster.

What the AI Evolved That the Engineers Did Not Write

One of the more instructive aspects of HarnessPro is the examination of what the evolution process actually produced. The team inspected the source code of evolved harnesses and found three mechanisms that they report were not explicitly encoded in the initial harness design — they emerged through the multi-agent RSI loop.

The first is dynamic routing. In an unoptimized harness, the same configuration — reasoning intensity, tool-call budget, context management strategy — applies to every incoming task regardless of complexity. The evolved harness instead reads the task description before execution and routes it to one of five configuration tiers: compact, lean, standard, deep, or research. Each tier has different settings for reasoning effort and maximum tool calls. A compact-tier task uses minimal reasoning and few tool calls; a research-tier task allows extended reasoning and a large tool budget. This task-adaptive resource allocation means that simple tasks consume only the resources they require, rather than receiving the same allocation as the hardest tasks in the benchmark suite.

The second is what the paper calls runtime budget hooks, implemented in an evolved hook.py file. The mechanism enforces tool-call budgets at three escalating thresholds. When the agent reaches 14 tool calls in a session, hook.py appends a message to the response of the tool call: a notice that the agent is entering an "economy phase" and should consolidate its next steps, stop repeating identical queries, and merge verification passes. At 26 tool calls, a stronger warning injects: the agent has entered a "convergence phase" and should stop broad exploration and focus exclusively on finalizing an answer. At 44 tool calls, the hook performs a hard block — it intercepts the next tool call and returns a rejection message instructing the agent to submit its best answer based on current evidence.

The engineering detail that makes the hook mechanism particularly efficient: rather than injecting budget notices into the system prompt or user turn (which would break the prefix KV-cache for long sessions, increasing cost), the evolved harness appends the notices to the suffix of tool return values. The system prompt remains unchanged, allowing prompt-prefix caching to function normally across the session. This cache-preservation detail is the kind of optimization that engineers regularly implement manually; that the RSI process arrived at it independently is notable.

The third mechanism is log folding. Command-line tool output during terminal and repository tasks frequently contains thousands of repetitive lines: identical error messages repeated in loops, verbose build-system output, or compilation diagnostics that repeat a single pattern hundreds of times. The evolved harness detects consecutive repeated patterns in tool output and collapses them to a single line with a "[previous line repeated N more times]" annotation, similar in spirit to the RTK (Rust Token Killer) tool. It also truncates excessively long outputs to preserve only the opening lines and closing lines. Neither of these features was present in the initial harness template; both directly address the 2.51M-token log-dumping failure case that the Trajectory Analyzer identified.

Ablation experiments confirm that none of these three mechanisms transfers cleanly to RHO in isolation. When dynamic routing alone is transplanted to RHO, cost falls but pass rate drops. When budget hooks alone are added to RHO, a similar pattern appears. The paper attributes this to co-evolution: the mechanisms were developed jointly and function interdependently. Dynamic routing works because the budget hooks enforce the allocations the routing layer defines; log folding works because the routing layer avoids allocating heavy reasoning to tasks where folded output would discard crucial diagnostic information. Separating them produces partial benefits but loses the synergy.

Where HarnessPro Stands in the Expanding RSI Landscape

The harness RSI subfield has accelerated substantially through 2026. RHO from Microsoft Research (June 2026) established the label-free, trajectory-based optimization paradigm that HarnessPro extends. RHI from Sakana AI and UC Berkeley (July 2026) approaches the problem at a different layer of abstraction: rather than modifying harness code directly, RHI evolves prompt-level specifications describing agent behavior. This difference in scope has practical consequences. Prompt-level specification evolution is lighter and faster to run than harness-code evolution, which requires isolated execution environments and Git commit cycles, but it cannot produce the kind of runtime-hook enforcement or structured log-folding logic that HarnessPro's code-level modifications generate. The two methods are complementary rather than directly competitive; a system that applies prompt-specification evolution followed by code-level harness evolution would cover both layers. The Harness-of-Harnesses framework (arXiv:2609.01481) takes a hierarchical approach, using one harness to supervise and improve another rather than running peer evolution. That design introduces a meta-level optimization loop but does not address the Pareto multi-objective selection problem that HarnessPro targets.

HarnessPro occupies a specific position in this space: it operates at the level of executable harness code (like RHO, unlike RHI), it is label-free (like RHO and RHI, unlike fully supervised variants), and it is the first system in this lineage to apply Pareto multi-objective optimization rather than single-metric optimization. The multi-tier output (Lite/Medium/Heavy from one evolution run) has no direct equivalent in prior work, where evolution produced a single configuration tuned for pass rate.

This Pareto-product-line output is arguably the most consequential contribution for deployment practice. A team deploying HarnessPro to production does not receive a single evolved harness and must choose between pass rate and cost by re-running evolution with different hyperparameters. Instead, a single evolution run produces three configurations that have been simultaneously co-optimized from the same evolution pool, each representing a verified Pareto-optimal operating point. The Medium configuration is effectively a cost-reduced drop-in replacement for RHO: it delivers similar pass rates at 53% lower spend. The Heavy configuration is for workflows where accuracy is the dominant constraint and cost is secondary. The Lite configuration enables cost-sensitive applications that need pass-rate improvement over the unoptimized baseline but cannot afford the full cost of state-of-the-art accuracy.

Several important limitations apply to these claims. All performance figures come from the HarnessPro paper itself; no independent research group has reproduced the results at the time of this writing. The paper's authors acknowledge that models trained on broad internet data may have encountered related repository patterns in pretraining, and while harness-only modification does not involve model fine-tuning, the underlying models' pretraining data remains a variable the HarnessPro paper cannot fully control for. The evolved harnesses are also specific to the model pairing used during evolution — configurations evolved for GPT-5.5 + Codex are not portable to a GPT-5.5 + Claude Code combination without re-running evolution, since the harness's dynamic routing thresholds, tool-call budgets, and log-folding parameters are calibrated to the latency and token-consumption characteristics of a specific model pair.

The evolution process itself requires running multiple training-set tasks across multiple rollouts per epoch, which has its own cost. The paper does not report amortized total spend including evolution overhead, which matters significantly for teams considering whether to run their own evolution loop or use the open-sourced evolved configurations directly against a compatible model pair. For teams already running GPT-5.5 + Codex or GLM-5.2 + Claude Code on SWE-Bench-style tasks, the pre-evolved configurations are directly usable. For teams running different model combinations, the evolution infrastructure — four specialized agents, Git versioning, isolated execution environments, multi-rollout evaluation — adds non-trivial engineering overhead before any cost savings accrue.

The evolved configurations released on GitHub represent pre-run evolution results for the specific models tested. Teams using different model versions or combinations would need to run their own evolution epochs, and the API cost of running three rollouts per candidate across a multi-epoch evolution loop could represent a significant upfront expenditure against which the per-task savings must be amortized over a sufficient production run volume.

The Pareto Frontier as a New Design Primitive for Agent Infrastructure

HarnessPro represents a concrete step toward treating agent deployment as a multi-objective engineering problem rather than a single-metric optimization race. The framing of agent performance exclusively through benchmark leaderboard position — which single-objective RSI naturally encourages — creates pressure to evolve in directions that optimize benchmark scores without regard for operational cost. Pareto-based selection builds cost visibility into the evolution loop from the beginning, preventing the selection dynamics from routing all evolved candidates toward expensive, high-accuracy configurations.

For the wider research community, the Test Synthesizer result (failure recall from 41.7% to 83.3%) sets a concrete target for evaluation methodology: label-free RSI research that continues using LLM-as-judge is operating on a signal that misses more than half of its failures. The improvement suggests that task-description-derived unit tests should become a standard component of label-free evaluation pipelines wherever task success can be expressed as deterministic code-level assertions. This has implications beyond the RSI subfield. Any agent evaluation framework that relies on LLM-generated verdicts over execution traces — including automated red-teaming pipelines, capability evaluations, and agentic safety testing — faces the same positivity-bias problem that the Test Synthesizer addresses. Where task outcomes can be reduced to executable assertions, generated test suites are a more reliable signal than model-generated text judgments.

The ContextPilot paper from lead author Zhuoshi Pan, accepted at EMNLP 2026 according to the HarnessPro paper's author biography, provides an adjacent technical thread. ContextPilot uses fine-grained reinforcement learning to train agents to manage context actively — deciding what to retain and what to discard in the context window. That work and HarnessPro address the token-efficiency problem from different angles: ContextPilot operates at the model-behavior level through RL training, while HarnessPro operates at the scaffolding level through harness code evolution. Combining both approaches — a model trained for context efficiency running inside an evolved harness that enforces budget discipline — represents an open research direction not yet explored in either paper.

What matters next for this framework is independent reproduction. The benchmark claims — particularly the 80.8% pass rate on Terminal-Bench 2 and the 53% cost reduction on SWE-Bench Pro — need external verification before they can be treated as established facts rather than promising findings. The open-source code release on GitHub makes that reproduction possible, and the community's response will determine whether HarnessPro's Pareto-based architecture becomes the foundation for the next generation of agent-optimization frameworks or a research prototype awaiting broader validation.