Zyphra unveils zha, a new sparse attention mechanism aiming for subquadratic scaling

Zyphra has introduced Zyphra Hierarchical Attention (ZHA), a novel sparse attention method that utilizes a tree-structured indexer to address limitations in existing sparse attention techniques. Unlike methods where the indexer scans the entire context, ZHA makes the search process itself sparse by compressing keys into hierarchical summaries. This approach allows the technique to be distilled into frozen pretrained models, such as Qwen3.6-35B-A3B, while maintaining performance close to dense attention on reasoning and coding benchmarks. Testing on Qwen3.6-35B-A3B showed that ZHA maintained high accuracy on retrieval tasks, staying above 93% on RULER-13, significantly outperforming MiniMax Sparse Attention (MSA) at 77.1%. In latency modeling, ZHA demonstrated substantial speedups, projecting gains of 373 to 417x over dense attention at 1 billion tokens. The method involves splitting keys into chunks, summarizing them, and using a top-p mechanism to adaptively select branches during decoding. While the base model remains frozen, only the reference queries are trained using a KL loss. Zyphra noted that results improved sharply when converting models of 4 billion parameters or more, and they also trained an MSA indexer on 40 billion tokens.