Multi-Agent LLM Systems Fail Compute-Matched Benchmarks 74% of the Time, New Synthesis Finds
A PACE protocol analysis of nine studies finds multi-agent gains shrink to near zero once compute is matched.

A new synthesis published Monday on arXiv concludes that most published evidence for multi-agent large language model superiority dissolves when the single-agent comparison receives an equivalent inference budget — a methodological flaw affecting 516 evaluated configurations across nine empirical studies.
The paper, "More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs," proposes a four-tier evaluation framework — PACE, for Parity-Aware Compute Evaluation — to separate genuine coordination gains from the simpler effect of running more inference passes.
The Greedy Baseline Problem at the Core of Multi-Agent Research
The central argument is straightforward. A multi-agent system makes multiple LLM inference calls — once per agent, often iterating — but the single-agent comparison in most published evaluations is a model responding once, using greedy decoding: selecting the highest-probability token at each step and producing one deterministic response. Comparing ten inference calls against one greedy pass does not test whether agents help each other reason. It tests whether more compute helps.
Across the nine studies and 474 matched-baseline contrasts, multi-agent systems equaled or fell below the single-agent baseline in 74 percent of cases. When sample count or inference budget was equalized, the mean accuracy difference was −0.042 — meaning multi-agent systems were, on average, slightly less accurate than a single model given the same resources.
How the PACE Framework Separates Coordination From Compute
PACE organizes baselines into four categories that form a ladder of increasingly fair comparisons.
Category A — greedy single pass — is the most common baseline in published work and the least informative. Category B — sample-matched — gives the single agent the same number of independent inference draws the multi-agent system uses. Under this condition, the mean accuracy advantage for multi-agent systems dropped by 0.038 across five studies that reported both comparisons. This "sampling gap" quantifies how much of the apparent multi-agent benefit was actually just extra samples.
Category C — resource-budget-matched — equates token expenditure or monetary cost, the strongest practical comparison for engineering teams. Category D — Pareto frontier — sweeps the full compute-budget range and asks whether multi-agent coordination ever dominates single-agent sampling at any cost level. For most benchmark-and-model combinations in the nine studies, it did not.
The framework also defines a resource vector with six dimensions: reasoning tokens, total tokens, monetary cost, latency, sample count, and scaffolding overhead. Latency deserves a note: none of the nine studies reported it — a gap that matters for teams deploying systems at scale.
When Extra Agents Genuinely Help — and the Narrow Evidence For It
The paper is not a blanket argument against multi-agent systems. It identifies conditions where coordination could produce genuine gains that more tokens alone cannot replicate: partitioned evidence, where each agent holds different information; external verification, where one agent fact-checks another using a tool unavailable during the original generation; and error diversity, where heterogeneous models fail on structurally different problem types.
These are framed as hypotheses, not tested findings. The closest empirical exception is narrow: on MMLU-Pro using Llama-3.1-70B, debate and mixture-of-agents exceeded ten-sample self-consistency by roughly two percentage points under matched compute. That result involves one benchmark and one model family, and an independent analysis of MMLU-Pro under matched compute found the same pattern to be non-reproducible across model scales.
The Resource Overhead Published Benchmarks Do Not Report
A parallel 2026 study comparing agentic and non-agentic configurations across five software-engineering tasks found that multi-agent designs consumed an average of 6.36 times as much energy and ran 6.07 times longer — with worst-case slowdowns reaching 160 times on individual task-hardware pairs. Fifty-nine of 66 Pareto-optimal configurations in that study were non-agentic or single-agent. The standard multi-agent evaluation pipeline — high-inference architectures against minimal single-pass baselines — systematically obscures both the accuracy and resource costs of deploying coordination at scale.
Limitations of the Synthesis and the Evidence It Does Not Settle
The paper is explicit about its own constraints. The nine-study corpus was not assembled through a systematic review: five studies were selected initially and four added after screening. One study, Zhang et al., contributes 180 of the 516 configurations, so aggregate statistics are sensitive to its inclusion. Excluding Zhang et al. moves the matched-baseline mean difference from −0.042 to −0.029 — still negative, but materially smaller.
The multilevel model rests on eight study clusters, which is underpowered for strong causal inference, and the intra-cluster correlation it assumes lacks empirical support in this corpus. These limitations do not overturn the directional finding, but the precise figures should be read as estimates rather than settled constants.
What Reviewers and Engineering Teams Can Do Now
The paper releases a PACE Reviewer Checklist — structured questions that program committees can apply to any multi-agent LLM submission: did the baseline receive matched samples, were inference costs reported, was latency measured, was the Pareto frontier traced? The code and checklist are publicly available.
For engineering teams, the implication is direct: before committing to a multi-agent orchestration architecture, test whether a single model running repeated sampling or extended chain-of-thought reasoning achieves equivalent accuracy at lower cost. The synthesis makes the burden of proof explicit — multi-agent proponents must now demonstrate that coordination adds something the inference budget alone cannot explain. Whether peer review norms shift accordingly depends on how much of the field's tooling and commercial investment has already been built around the assumption that more agents means better performance.



