IBM, Supermicro, and NVIDIA Validate KV Cache Storage That Cuts Inference Latency by 56x

A joint validation by IBM, Supermicro, and NVIDIA has demonstrated that routing transformer attention data to enterprise shared storage — rather than recomputing it on the GPU — can cut first-token latency by 56x for 130,000-token contexts and boost concurrent throughput by 22x. The finding, documented in a co-authored IBM Redbook published earlier this year and formally announced today ahead of Supercomputing 2026 (SC26), offers AI infrastructure teams a concrete blueprint for scaling large-language-model inference without proportionally expanding GPU fleets. The 19-engineer authorship team across all three companies published hardware configurations, software versions, and benchmark results in enough detail to serve as an enterprise procurement guide.

The announcement lands ten days before SC26 opens in Chicago — the AI and HPC industry's annual hardware showcase — where all three companies are expected to demonstrate the architecture. The formal launch of the reference design is a co-branding event for a technical document that has been publicly available since June 2026, when the IBM Redbook titled "Context Without Limits" was first published and later updated in August.

Why GPU HBM Runs Out During Long-Context Inference

Understanding what the architecture solves requires understanding why GPU memory exhausts in the first place. When an LLM processes a prompt, it computes key (K) and value (V) projections for every token in its context window through each transformer attention layer. Those projections are retained in a cache — the KV cache — so that subsequent tokens in the same response can access them without repeating the computation. This is efficient by design: recomputing would waste thousands of GPU operations per token.

The problem is size. KV cache storage requirements scale with the number of model layers, attention heads, context length, concurrent requests, and data type precision simultaneously. A 120-billion-parameter model handling 50 concurrent sessions at 128,000-token contexts can require more than a terabyte of KV data — far beyond the 80 to 192 gigabytes of high-bandwidth memory (HBM) on a single data-center GPU. When HBM fills, inference engines face an uncomfortable set of choices: truncate the context (losing information), evict and recompute (expensive and slow), or queue requests (reducing throughput).

The IBM/Supermicro/NVIDIA reference architecture adds a fourth option: move cold KV cache blocks to a network-attached storage tier and retrieve them on demand. The benchmark result captures the before-and-after contrast directly: recomputing a 130,000-token cache from scratch took 32.14 seconds of GPU time. Retrieving the same data from the shared storage tier took between 0.19 and 0.57 seconds — with timing that remained flat regardless of context length, because the retrieval time depends on data size and network speed, not on model computation, according to the IBM Redbook Context Without Limits.

NVIDIA Dynamo's Tiered Storage Architecture and the KVBM/NIXL Layer

The core enabling technology on the software side is NVIDIA Dynamo, an open-source distributed inference framework released in production form at GTC 2026. Dynamo organizes KV cache storage into a five-level hierarchy: G1 (GPU HBM, microsecond access), G2 (system DRAM, millisecond), G3 (local NVMe), G3.5 (NVIDIA's CMX network-attached appliance), and G4 (shared enterprise storage accessed over the network).

The KV Block Manager (KVBM) is Dynamo's internal component responsible for allocating KV blocks across these tiers and tracking which blocks are "hot" (recently accessed, kept in upper tiers) versus "cold" (candidates for eviction to lower tiers). The NVIDIA Inference eXchange Library (NIXL) handles the actual data transport — asynchronously moving blocks between tiers without stalling the inference engine. Where supported hardware is available, NIXL can use RDMA or NVIDIA's GPUDirect Storage technology to transfer data directly between storage and GPU memory without routing through the CPU, reducing both latency and CPU overhead on the inference host.

The IBM/Supermicro validation placed IBM Storage Scale ECE at the G4 tier. Storage Scale ECE is IBM's software-defined parallel filesystem — the enterprise edition of IBM GPFS, used in many of the world's largest supercomputers for genomics, financial modeling, and media workflows. For the inference use case, it provides RDMA support and GPUDirect Storage compatibility over Ethernet, meaning NIXL can stream KV blocks directly into GPU memory from Storage Scale without CPU involvement.

Hardware Stack and Cluster Configuration

The reference architecture's storage layer uses eight Supermicro ASG-1115S-NE316R Petascale Storage Servers, each a 1U chassis built around an AMD EPYC 9535 (Turin) processor. The choice of AMD's EPYC Turin is deliberate: the 9535 provides 128 PCIe Gen5 lanes, split evenly between 16 Micron E3 7.68 TB NVMe drives (in E3.S form factor) and two 400 Gb/s network ports. That lane allocation avoids the PCIe bandwidth contention that can occur when NVMe and high-speed networking share a narrower bus.

The network fabric is NVIDIA Spectrum-X, using three SN5600 switches in a spine-leaf configuration. Spectrum-X is NVIDIA's AI-purpose-built Ethernet fabric — it includes adaptive routing and congestion control tuned for the many-to-many traffic patterns of AI all-reduce operations and storage access, distinguishing it from general-purpose datacenter switches. Each storage server connects at dual 400 Gb/s (800 Gb/s aggregate per node), with the eight-node cluster providing up to 6.4 Tb/s of internal network bandwidth.

The inference test node used four NVIDIA RTX PRO 6000 Blackwell workstation GPUs on a Supermicro SYS-212GB-FNR server with an Intel Xeon 8592 CPU. The software stack was IBM Storage Scale ECE v6.0.0.1, NVIDIA Dynamo v0.9.0, vLLM v0.14.1 as the inference engine, and NVIDIA AIPerf v0.6.0 as the load generator. The model was openai/gpt-oss-120b, quantized to MXFP4 — NVIDIA's 4-bit microscaling format introduced with Blackwell — which reduces memory footprint compared to BF16 while preserving throughput.

For cluster sizing, the Redbook defines three reference tiers: a small configuration (4 storage nodes, up to 64 GPUs, 20–75 GB/s read bandwidth, targeting 7B–34B models); a medium configuration matching the benchmark (8 nodes, 100–300 GB/s, 100–256 GPUs, 70B–120B models); and a large configuration (16+ nodes, 300+ GB/s, 1,000+ GPUs, multi-model deployments above 120B parameters).

Benchmark Results and What They Reveal About the Architecture's Ceiling

The validation produced three distinct benchmark results. In the first, the team measured TTFT (time to first token) for single-GPU inference at varying context lengths, comparing GPU recomputation against G4 retrieval. At 32,000 tokens, recompute took approximately 7 seconds; at 130,000 tokens, it reached 32.14 seconds. The G4 retrieval path returned results in 0.193 to 0.57 seconds across all context lengths — the flat curve is a defining characteristic of the retrieval approach, since the fetch time is bounded by network latency and data size rather than model computation. The 56x speedup figure comes from comparing the worst-case recompute (130K tokens) against G4 retrieval, as detailed in the IBM Redbook Context Without Limits.

The second benchmark measured concurrent throughput: 28 simultaneous connections, 100 unique prompts, totaling 24 million tokens and 825 GB of KV cache data moved. Baseline throughput with no cache offload was 0.19 requests per second, constrained by the GPU spending most of its time recomputing evicted context. With G4 offloading active, throughput reached 4.26 RPS — a 22x increase — and 200 requests completed in 46.94 seconds compared to 1,048 seconds without the offload layer.

The third benchmark simulated a noisy-neighbor environment: a competing network workload pushing 200 GB/s of additional traffic across the same Spectrum-X fabric. Under that stress, the G4-enabled configuration sustained 3.6 RPS, compared to 0.19 RPS baseline — still an 18x improvement — but with an 18% throughput reduction compared to the clean-network case. This is a meaningful real-world test for multi-tenant enterprise deployments, where storage networks are shared resources.

An important detail buried in the methodology: G1 (GPU HBM) and G2 (system DRAM) KV cache tiers were disabled entirely for all three benchmarks, with capacity set to zero. The tests measured G4-tier performance in isolation, not a mixed-tier hierarchy as would typically be deployed. That is both a strength (it characterizes the G4 tier independently) and a limitation (production systems will generally use all tiers together, with G4 handling only the coldest, most infrequently accessed blocks, which would yield different performance characteristics).

The more structurally significant finding is the gap between storage throughput and actual network utilization: the eight-node cluster delivered 315 GB/s of sustained read bandwidth, but the benchmark consumed only 80 GB/s — the client's dual 400 Gb/s NIC (representing 160 GB/s theoretical maximum per client, with overhead reducing that further) was the actual ceiling. The storage system had nearly 4x headroom. For teams planning to scale beyond the benchmark configuration, the upgrade path runs through client networking — either denser GPU nodes per storage cluster, or moving to 800 Gb/s client NICs — before additional storage nodes become necessary.

Where the IBM/Supermicro Approach Sits in a Crowded KV Cache Storage Market

The enterprise KV cache offload market is developing rapidly, with multiple vendors targeting the same GPU-memory bottleneck from different architectural directions.

VAST Data has positioned its unified flash platform as an AI storage layer, reporting a 9.7x throughput improvement in a benchmark with AMD Instinct MI355X hardware. WEKA's Augmented Memory Grid targets the cluster-local tier, aiming to maximize active-session KV cache bandwidth within a rack. Dell has combined its PowerScale file storage with the open-source LMCache library, publishing a 19x time-to-first-token improvement in its own validation at 131,000-token contexts.

The IBM Storage Scale differentiation centers on the G4 tier's persistence properties. Local NVMe approaches (G3 and WEKA-style architectures) provide fast retrieval for cached context within a single node or rack, but the cache is lost when a session migrates to a different node or when a server restarts. A shared parallel filesystem like Storage Scale can persist KV cache data across days, weeks, or months, making previously computed context reusable across sessions, models, and hardware upgrades. For enterprise agentic workflows that maintain persistent conversation histories or regulatory audit trails, the difference between ephemeral and durable KV storage is significant.

The noisy-neighbor benchmark is also a differentiator in the competition. Most vendor KV cache benchmarks test clean isolated environments. The IBM Redbook's explicit inclusion of a competing-workload test acknowledges the reality of multi-tenant enterprise deployments, where inference clusters share networking with backup systems, monitoring agents, and other storage consumers.

Limitations, Open Questions, and What to Watch at SC26

Several constraints deserve direct attention. The benchmark's GPU hardware — four RTX PRO 6000 Blackwell workstation GPUs — is not representative of data-center inference configurations. Production deployments at this model scale would use NVIDIA H100, H200, or B200 GPUs, with significantly larger HBM capacity and different PCIe topologies. The reference architecture's sizing tables are extrapolated to those configurations, but no benchmark data with data-center GPUs is published in the Redbook.

The benchmark model, identified as openai/gpt-oss-120b at MXFP4 quantization, is not a widely distributed standard reference model. Its provenance and relationship to other 120B-parameter models are not explained in the Redbook, which limits comparability to other published inference benchmarks.

The KV cache tenant isolation question is not addressed in the Redbook for the G4 shared storage tier. When multiple tenants' KV cache blocks reside on the same Storage Scale cluster, the access control and isolation model — which blocks can be accessed by which inference engine, and what happens if blocks are incorrectly attributed or evicted by one tenant's workload affecting another — requires explicit security design. NVIDIA's Inference Context Memory Storage Platform (ICMSP), announced at CES 2026, addresses tenant isolation using BlueField-4 DPUs at the network edge, but this Supermicro/IBM configuration uses Spectrum-X and does not specifically describe BlueField DPU-based isolation between tenants.

No independent benchmark replication exists as of this writing. All quantitative claims come from the vendor-coordinated authorship team. That is not unusual for a reference architecture publication — but independent reproduction of the 56x and 22x results, using data-center GPUs, would significantly strengthen the case.

SC26 opens in Chicago on November 15 and runs through November 20, with exhibits open November 17–19. All three companies are expected to present the architecture on the show floor. SC26 historically surfaces independent HPC and AI infrastructure evaluations alongside vendor demonstrations, and practitioners who attend will be in a position to assess whether the reference architecture's performance translates from a controlled 8-node lab test to production deployments at larger scale.

For enterprise AI teams evaluating inference infrastructure for large models today, the IBM Redbook provides one of the most detailed public engineering documents available for a G4 KV cache tier — a rarity in a market where vendors typically publish headline benchmarks without full methodology. The network-bandwidth ceiling identified in the validation data may prove to be as useful as the throughput headline: it tells teams where to look when their KV cache offload architecture stops scaling.