Google launches llm-d 1.0 with cache-aware routing for distributed inference

Google Cloud has released llm-d 1.0, an open-source, Kubernetes-native layer designed to coordinate vLLM or SGLang model servers into a unified pool. The project, launched at the PyTorch Conference North America, targets self-hosted agent and reinforcement learning workloads where traditional load balancing fails. llm-d, founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, is a CNCF Sandbox project accepted on March 12, 2026, and licensed under Apache 2.0. It features prefix-aware routing, KV cache offloading, and batch inference, with IBM emphasizing its role in day 2 operations. Adopters report up to 75-80% cache hit rates, 4x throughput increases, and 3,000 concurrent coding agents at 5-10 times lower API pricing. Google Cloud offers GKE Inference Gateway as a managed service, supporting global failover and auto-scaling. The project collaborates with CoreWeave and NVIDIA, integrating Dynamo components into llm-d.