Bytedance's seed team explains AI performance fluctuations in long text processing

ByteDance's Seed team has published a research paper detailing the technical causes behind performance fluctuations in large language models when handling ultra-long text. The study identifies 'phase sensitivity' as the primary factor, stemming from block-based KV cache compression technology. This method reduces memory usage by compressing continuous token windows into fewer entries with fixed strides. However, it introduces a new positional coordinate—the 'phase' of a token relative to the compressed window boundaries. Experiments revealed that identical information can lead to significant retrieval difficulty differences based on phase variations. In some large open-source models, phase differences caused a 40 percentage point drop in long-text retrieval accuracy. The findings underscore the limitations of traditional average benchmark tests, which may mask periodic weaknesses in models' performance. As long-text applications grow, this research offers critical insights for future model architecture improvements and enhanced long-context reasoning stability. The research highlights the need for more rigorous testing methods to address hidden performance issues in large models. The Seed team's work provides a theoretical foundation for optimizing model stability and efficiency in long-context reasoning tasks. This development is significant as the demand for robust AI performance in extended text processing continues to rise.