Answer-Specific Verifiers Beat Majority Voting for LLM Output Selection

A single-author arXiv preprint submitted October 19 demonstrates that routing to a verifier that reads an LLM's actual candidate answer outperforms majority voting for selecting correct outputs across five tested benchmark settings. The result, by Ziqian Zhang, provides both a Lean 4–certified theoretical bound and empirical validation that the answer text carries failure-detection information that prior-free selection strategies cannot recover — directly challenging majority voting's role as the default output-aggregation strategy in production inference pipelines.

The practical difference is substantial. On GSM8K using Mistral-7B with eight draws, an external answer-reading verifier named Jev selects a correct answer on 62.2 percent of tasks, compared with 44.8 percent for majority voting — a gain of 17.4 percentage points (p = 0.0025). Across all five benchmark settings, Jev outperforms majority voting by margins of +3.0 to +17.4 percentage points.

All figures are author-reported from an unreviewed preprint; no independent reproduction exists at time of publication.

Read more: InternLM open-sources multimodal decision models that outperform Jev

Pooled AUROC Mixes Task Difficulty With Answer-Specific Signal

The paper's theoretical contribution identifies a structural problem with how verifiers are currently evaluated. Any failure score can encode two distinct types of information: how hard the task is regardless of which answer was given, and whether this specific answer is correct. Standard pooled AUROC conflates these signals, because most failing–passing comparisons in a pooled evaluation are drawn from different tasks. A verifier that simply reflects task difficulty — without reading the answer at all — appears discriminative in pooled metrics, because harder tasks fail more often.

Zhang formalizes this problem by introducing within-task AUROC: a metric computed only over failing and passing draws of the same task. By construction, a score that is constant within a task yields a within-task AUROC of exactly 0.50, revealing that it carries zero answer-specific information. To certify the ceiling formally, the paper uses Lean 4, a proof assistant used in formal mathematics, to machine-verify the maximum within-task draw-level AUROC that any task-only (question-only) ranking can achieve — establishing a guaranteed bound on what answer-reading adds.

The framework extends Bulut's 2026 "It's the Problem, Not the Path", which introduced within-problem evaluation as a diagnostic for pooled AUROC inflation, by adding the Lean 4–certified ceiling and the selection application.

How the Verifiers Work and What They Achieve

The paper tests two architectures. Jev, the primary verifier, is a hosted external judge that receives a question and a proposed answer, then returns a score reflecting the probability the answer is wrong. Its training data is not publicly documented, which limits external auditing of its benchmark validity.

The second is a self-trained alternative: a Qwen3-0.6B model fine-tuned as a sequence classifier on question–answer pairs, accepting up to 1,536 tokens. This verifier is cheaper and fully under experimenter control, though it requires labeled examples for each new domain.

Across five benchmark settings — GSM8K with reused tasks, GSM8K with new tasks, MATH, AutoLogi, and EvalPlus — Jev achieves within-task AUROC of 0.837 to 0.904. A second external judge improves failure-prediction AUROC by 0.132 to 0.240 points over prediction from the question alone, confirming that the answer text is the load-bearing signal rather than surface features of the question.

Where Answer-Specific Verification Hits Structural Limits

Three constraints cap the approach's practical gains. The first is coverage: if none of the N draws contains a correct answer, the verifier still fails. On GSM8K, 71 of 148 tasks had no correct candidate among eight draws — a problem that better base models or larger sampling budgets address separately from selection quality.

The second is saturation. On EvalPlus, 99 of 103 tasks already succeed under majority voting, leaving almost no headroom. The +3.4-point improvement there reflects a ceiling effect, not a weak verifier.

The third is cost. Eight draws plus eight Jev judgments cost approximately as much as 11.4 unjudged draws. The verifier wins in these settings, but the cost ratio narrows quickly on distributions where majority voting already succeeds at high rates.

What This Means for AI Systems That Check Their Own Outputs

The result carries a direct implication for agentic systems. Any pipeline that samples multiple outputs and votes forfeits answer-specific signal that exists at no extra cost once the response is generated. Zhang formalizes this loss and shows recovery requires only a verifier that reads the answer.

The broader verifier ecosystem is now engaging with whether within-task discrimination, rather than aggregate accuracy, is the right design target. The Lean 4–certified ceiling gives that design conversation a formal anchor: any verifier that does not read the answer is provably bounded, and the bound is calculable. As open-weight alternatives to Jev mature and cost-parity research reduces the judgment overhead, the tradeoff Zhang documents will shift — the question is whether discrimination gains can hold when the verification budget narrows.

Read more: NUS open-sources Jeva, an open-weight alternative in the Jev ecosystem