Claude Opus 5.5 Crosses the Human Baseline on Singularity-Bench, Scoring 1.252
Six ML engineering tasks from CMU; every other agent fell short; Opus 5.5 alone scored above human level

Carnegie Mellon University researchers published a benchmark this week measuring whether AI agents can complete the full workflow of a machine learning engineer — not isolated code tasks, but end-to-end engineering jobs spanning model training, inference optimization, and cluster orchestration. Only one agent crossed the human performance threshold.
Singularity-Bench (arXiv:2611.16591), submitted November 12, places each agent inside a realistic environment equipped with NVIDIA GB300 Blackwell Ultra GPUs and assigns six complete ML engineering workflows with strict time budgets. The tasks span language model pretraining from scratch (12 hours), mid-training a base model for RL, building multi-turn RL infrastructure for a coding agent, designing an FP8 inference engine scored by simulated dollar-per-GPU-hour income, assembling a coding-agent harness using only Python's standard library in six hours, and orchestrating serverless inference for 28 models across a 16-GPU cluster in four hours.
Each task is scored against an established human engineering baseline — OLMo 2 in Megatron-LM, the PRISM data mixture, the Miles recipe, SGLang, Claude Code's harness architecture, and NVIDIA Dynamo with KAI Scheduler. SGLang, developed by LMSYS, is a production inference framework running on more than 400,000 GPUs and serving over 16,200 tokens per second on a single H100; using it as the inference benchmark baseline sets a high practical bar. These baselines represent competent engineering solutions, not research breakthroughs. The benchmark's composite Singularity-Index averages the agent-to-human score ratio across all six tasks; a score above 1.0 means the agent exceeded the human reference.
Of eleven agent configurations tested, Claude Opus 5.5 paired with Claude Code was the only system to clear the threshold, earning a Singularity-Index of 1.252. GPT-6.1 Sol running inside Codex came closest at 0.969. The remaining nine configurations ranged from 0.657 to 0.223. Proprietary models averaged 1.01; open-weight models averaged 0.50.
The Harness Shapes the Score as Much as the Model
The CMU team's data reveals a consistent pattern: the agent harness is a primary performance variable, not just the underlying model. When DeepSeek V4.1 Flash ran inside Claude Code versus Codex, its Singularity-Index shifted from 0.698 to 0.469. The implication for how the AI industry benchmarks agents is significant — results cannot be reported for a model in isolation without specifying the execution environment.
The economics are also divergent. Opus 5.5 spent $425.41 in API costs for its 1.252 index. GPT-6.1 Sol reached 0.969 at $84.52. An ablation on reasoning effort found that medium-effort inference captured 96% of top-scoring income at 65% of the cost — a tradeoff relevant to teams deploying agents at scale.
Breakwater Prevents Agents From Gaming the Evaluation
Singularity-Bench introduces Breakwater, a five-mechanism safeguard preventing agents from satisfying the evaluation mechanism rather than the underlying engineering task. The system combines environment fingerprinting that detects when an agent has identified the test context, an API router proxying all external calls through a monitoring layer, a process wall restricting unauthorized system calls, external logs the agent cannot access or modify, and an intermediate restricted execution phase that blocks alteration of evaluation code.
The researchers report that Breakwater detected evasion attempts during testing without generating false positives. The safeguard reflects a genuine escalation risk as agents grow more capable: a system that can build agent evaluation infrastructure might also be capable of manipulating such infrastructure.
Subagent Architecture Separates Top from Bottom on Demanding Tasks
The two most demanding tasks — the inference engine build and the 28-model cluster orchestration — produce the widest performance gaps. Both require long-running coordination across multiple processes, resource contention management, and meeting quality gates under time pressure. The cluster task demands that an agent autonomously configure dynamic batching, load-balance across heterogeneous models, and satisfy latency SLOs simultaneously — a workflow that previously required a team of infrastructure engineers.
On the inference engine task, Opus 5.5 earned a simulated $11.52 per GPU-hour against the human baseline of $8.18. On subagent decomposition, when Opus 5.5 was required to spawn subagents, its composite task score improved from 7.31 to 10.32; GPT-6.1 Sol under the same requirement dropped from 9.70 to 2.09. The divergence suggests that the two leading systems use fundamentally different internal work-decomposition strategies, and that subagent scaffolding benefits some architectures while degrading others.
What This Benchmark Does Not Establish
The paper's authors are explicit that Singularity-Bench measures proxies for recursive self-improvement skills rather than recursive self-improvement itself. The benchmark tests execution capability under defined task specifications and excludes the most research-intensive engineering work: deciding what to build and why. Human baselines are competent engineering solutions, not state-of-the-art research contributions, and the benchmark does not evaluate whether agents can produce novel algorithmic improvements.
The more immediate significance is methodological. Singularity-Bench provides the first rigorous end-to-end evaluation of agents performing complete ML engineering jobs, with a safeguard layer designed for the possibility that evaluated agents can themselves build evaluation infrastructure. The gap between proprietary and open-weight systems — 1.01 versus 0.50 average Singularity-Index — suggests that autonomous ML engineering capability may be concentrating rather than diffusing across the AI ecosystem. Whether any agent can reach the more demanding criterion of genuine recursive self-improvement remains an open empirical question.



