Nvidia addresses AI memory bottleneck with full-stack design, boosting per-dollar throughput by 30x

Nvidia recently detailed a strategy to overcome mounting GPU memory pressures caused by the rapid expansion of large models and agent applications. Recognizing that HBM supply constraints and rising costs will persist, the company introduced a concept of full-stack collaborative design, integrating GPU, CPU, networking, storage, and software to maximize the value of every byte of memory. This architecture addresses the memory wall issue—where data transfer speed limits capacity gains—by optimizing data placement. Specific implementations include using NVFP4 low-precision format for compressing model weights and KV caches, distributing model weights across up to 72 GPUs in the NVL72 system, and leveraging Vera CPU's large LPDDR memory for context buffering. Benchmarking results from SemiAnalysis AgentX show that the Vera Rubin NVL72 achieves a 30-fold increase in per-dollar agent throughput compared to the GB300 NVL72, despite only a modest 3% increase in HBM capacity per GPU. Furthermore, the GB300 NVL72 demonstrated a 10-fold throughput improvement per dollar over the H200 platform under specific conditions, signaling a shift in AI competition focus from sheer memory size to effective token generation per byte and watt.