Google unveils TPU sync: open-source tool for efficient kv cache and weight transfer on tpus

Google has released TPU Sync, an open-source C++ library designed to manage the movement of model weights and KV caches across various memory tiers within Cloud TPUs. This library treats TPU HBM, host DRAM, shared storage, and the data center network as a unified memory hierarchy, addressing the growing bottleneck of model state movement rather than computation. Internal testing conducted by Google in July 2026 on TPU v7x (2x2x1) showed that TPU Sync achieved over 90% lower KV transfer latency when handling MoE models ranging from 235B to 480B parameters. TPU Sync facilitates two primary functions: managing KV cache state during inference and broadcasting weights from trainers to samplers in reinforcement learning. The architecture involves a global KV store, a transfer controller, and a KV cache manager, where prefill nodes offload blocks to pinned host RAM via asynchronous DMA before they stream to decode nodes' HBM. The library is licensed under Apache-2.0 and integrates with PyTorch via TorchTPU and natively with JAX. Integrations are already established with vLLM-TorchTPU as the TPU KV connector and SGLang-TorchTPU as the P/D transfer backend. Furthermore, llm-d 1.0 on GKE utilizes it for prefix-cache-affinity routing and experimental spill to Google Cloud Managed Lustre. Meta and Wayfair are reportedly evaluating the architecture, while Inferact and RadixArk co-engineered the vLLM and SGLang integrations.