Google Cloud has released TPU Sync, an open-source C++ library for moving KV caches and model weights on Cloud TPUs. The Google TPU Sync KV cache transfer library treats TPU HBM, host DRAM, shared storage and the data center network as one memory hierarchy. It targets a growing bottleneck: moving model state, not computing it. Google reports over 90% lower KV transfer latency on 235B to 480B parameter MoE models.
TL;DR
- Library: Native C++ core with Python wheels for PyTorch and JAX. Tested on 235B to 480B MoE models.
- Runs on: Cloud TPU v7x (Ironwood) and v6e per the README. Benchmarks used TPU v7x (2x2x1), PCIe Gen5 and 200 Gbps NICs.
- Performance: Large cuts to KV transfer latency and time to first token (TTFT) in Google’s internal lab tests.
- Best: Up to 90% lower average TTFT (480B P/D serving at low concurrency, and 64K-context prefix caching).
- Worst: No RL numbers published. The README says it is not yet recommended for general use.
- Bottom line: Best: Qwen3 235B KV transfer drops from several seconds to ~230 ms. Worst: every number is self-reported and early-stage.
What is TPU Sync?
TPU Sync is an open-source C++ library from Google that moves TPU-tiled tensors across memory tiers with asynchronous DMA. It handles 2 jobs: KV cache state during inference, and trainer-to-sampler weight broadcast in reinforcement learning (RL). The repository is licensed Apache-2.0.
How does TPU Sync work?
The library spans TPU HBM, pinned host DRAM, shared storage and multi-host DCN. Storage uses experimental POSIX backends, including local SSDs and Google Cloud Managed Lustre. Google lists 4 design choices:
- Layout-preserving direct DMA: It reads JAX PJRT buffers or PyTorch tensors in place. Data keeps its native TPU tile layout, with no Python serialization or bounce buffers.
- Compute-overlapped transfers: Raw PJRT buffer aliases let transfers run during computation. Built-in DMA pacing stops large copies from delaying kernel launches.
- No XLA compilation in the data path: Transfers and resharding run in native C++ outside the XLA graph.
- High-bandwidth PCIe and DCN: It reaches up to 223 GB/s per Ironwood host, against a 256 GB/s PCIe Gen5 ceiling. DCN streaming hits ~192 Gbps on 1 NIC, or ~350 Gbps across 2.
Above this sit a global KV store for prefix lookups, a transfer controller and a KV cache manager.
How does disaggregated prefill/decode serving use it?
The prefill node offloads KV blocks to pinned host RAM through async DMA. The controller streams them host-to-host to the assigned decode node. Arriving blocks move into decode-node HBM while decoding continues.
‘}).join(”);setTimeout(function(){bars.querySelectorAll(‘.fill’).forEach(function(f){f.style.width=f.dataset.w+’%’})},40);post()}
R.querySelectorAll(‘.seg button’).forEach(function(b){b.onclick=function(){R.querySelectorAll(‘.seg button’).forEach(function(x){x.classList.remove(‘on’)});b.classList.add(‘on’);drawBars(b.dataset.s)}});
function gauges(){R.querySelectorAll(‘.g .num’).forEach(function(n){var to=+n.dataset.to,t0=null;function f(ts){if(!t0)t0=ts;var p=Math.min((ts-t0)/1100,1);n.textContent=Math.round(to*(1-Math.pow(1-p,3)));if(p<1)requestAnimationFrame(f)}requestAnimationFrame(f)});var pc=document.getElementById(‘pcie’);pc.style.width=”0″;setTimeout(function(){pc.style.width=(223/256*100).toFixed(1)+’%’},40)}
window.addEventListener(‘load’,post);window.addEventListener(‘resize’,post);setTimeout(post,300);
})();