Stanford and uc berkeley researchers introduce TokenGate to accelerate long-context prefill in llms

Stanford and UC Berkeley researchers have introduced TokenGate, a module designed to significantly speed up long-context prefill in large language models (LLMs). TokenGate employs token pruning to allow existing LLMs to process only the most relevant tokens, achieving a 1.4x to 11.3x improvement in time-to-first-token (TTFT) without sacrificing accuracy. The module also reduces peak prefill memory usage by 1.5x to 3.4x. TokenGate is a drop-in component that integrates trainable token pruning into pretrained models, enabling efficient processing of long inputs. The research team tested TokenGate on several benchmarks, including RepoBench-P, 2WikiMultihopQA, and NIAH, outperforming existing methods like FastKV and SlimInfer. The module uses a learnable gate to refine attention logits, allowing for end-to-end training. The team also noted that TokenGate’s performance is attributed to backbone co-adaptation. The code and checkpoints are available on GitHub under the Apache-2.0 license.