NVIDIA Research: AI Agents Hold Accuracy at 16K Tokens as Long Contexts Cost 7x More
Benchmark study finds frontier agents hold accuracy at 16K tokens; 256K contexts cost 7x more on GPUs
An NVIDIA research team has published a preprint arguing that frontier AI agents do not need long context windows to perform well on agentic benchmarks. Using two compaction strategies tested across seven language models and thirteen tasks, the paper reports that capping agent conversation history at 16,000 tokens can preserve or improve scores relative to unconstrained history accumulation — while reducing the KV-cache memory pressure that makes long-context inference expensive on shared GPU infrastructure.
The paper's headline figure — 6.5× to 54.6× cost savings — requires careful interpretation. Those numbers apply to a hypothetical estimated pricing model the authors call EORC (estimated optimized retail cost), which assumes short contexts cost roughly one-fifth as much as maximum-context serving. At published API prices, the picture is more varied: on SWE-bench Verified, the best compaction setting reduced Claude Opus 4.5's nominal API cost from $420 to $296 per benchmark run, a 1.4× reduction at standard pricing. The same setting is estimated under EORC to reduce cost to around $70 — a hypothetical that reflects infrastructure efficiency gains rather than current API list prices. For GPT 5.6 Sol at the same benchmark, compaction's overhead actually raised nominal cost compared with the full-history baseline. Infrastructure operators self-hosting open-weight models on H100 GPUs see the clearest benefit: the paper measures a 256K-token context costing 7× more per output token than a 16K context on Qwen 3 Coder Next, due to KV-cache memory and decode throughput constraints.
How the Three Context Regimes Work
The paper benchmarks three context regimes. In Full-History Accumulation (FHA), all prior turns accumulate without bound — the baseline most deployed agents use today. In End-of-Conversation Compaction (EoCC), when history reaches a configurable threshold (tested at 16K–48K tokens), the model writes a structured summary then restarts with a fresh conversation seeded only by that summary. The cycle repeats at each threshold crossing. In Single-Action Compaction (SAC), the agent takes exactly one tool action per conversation instance, writes an updated semantic state, and discards the entire history before the next step. The best compaction prompts identified by the paper are not narrative summaries but explicit structured handoffs containing three elements: a verbatim copy of the original task, the current plan and intentions, and all artifacts that future execution steps will need.
Frontier Agents Held Accuracy; Certain Open-Weight Models Did Not
On SWE-bench Verified, the best EoCC setting pushed Claude Opus 4.5 from 70.0% to 74.8% — a 4.8-percentage-point improvement while reducing median peak context from 32K to 19K tokens. GPT 5.6 Sol improved from 84.4% to 88.8% under its best EoCC setting. Single-Action Compaction also held or matched baseline on SWE-bench: Claude Opus 4.5 held at 70.2% under SAC, cutting median peak context to 8K tokens, though GPT 5.6 Sol saw a modest 2-percentage-point decline under SAC, to 82.4%.
Open-weight models produced sharply uneven results. Qwen 3 Coder Next fell from 53.0% to 3.6% under aggressive compaction — a near-total collapse the authors attribute to difficulty maintaining coherent state across compaction cycles at tight context budgets. MiniMax M2.7 was more robust, dropping only from 72.6% to 67.8%. These results mean compaction is not a universally safe strategy: model selection and threshold calibration matter significantly before any deployment.
Long-Input Gains Partly Reflect a Workflow Change
For long-input tasks — benchmarks like NIAH, RULER, and BABILong that test retrieval from large document sets — the approach differs from compaction. Instead of pasting full documents into context, the agent stores them as filesystem files and retrieves relevant content using grep or head commands. On NIAH (Needle-in-a-Haystack), this pushed Claude Opus 4.5 from 60% to 100%. The 54.6× cost figure for NIAH reflects this workflow substitution as much as any algorithmic compaction: the savings are real, but they represent a re-engineering of document access rather than a property of the compaction method.
Anthropic Already Ships Compaction; NVIDIA Outlines What May Come Next
The paper's techniques are not entirely novel in production. Anthropic has offered a context compaction API endpoint since January 2026, and Claude Code's auto-compact feature has been active since version 2.0.64. What the NVIDIA preprint adds is systematic empirical evidence across a wider range of frontier models, plus a conceptual proposal the authors call Semantic State Space Models (SSSMs): post-trained models that maintain an explicit natural-language state representation between turns rather than accumulating raw history. No SSSM implementation exists; it is a research direction, not a deployable system.
For teams running agents today, the practical implication is narrower than the headline savings suggest. API users at standard pricing should validate compaction threshold choices against their specific model and task before treating EORC figures as a planning baseline. The more durable finding may be simpler: across most tasks tested, models do not appear to need their entire prior history to perform well — a result that challenges assumptions baked into most agent frameworks and creates meaningful infrastructure headroom as agent deployments scale.