New benchmark reveals limitations of audio large models in remembering speech details

A joint team from the University of Melbourne and the University of New South Wales has introduced VoxMem, a benchmark designed to evaluate the ability of audio large models to remember who is speaking, along with tone and background sounds. The benchmark highlights two major flaws in current evaluation methods: they focus only on transcribed text and fail to isolate the impact of conversation length. VoxMem uses a two-dimensional classification method, covering 15 combinations of acoustic evidence and memory operations. It includes 3,196 evaluation instances and 34,743 audio conversations, totaling about 177 hours of audio. Testing 15 mainstream audio large models showed that none exceeded a 40% accuracy rate at a context length of around 32K. Models performed better at remembering what was said than speaker identity, tone, emotion, and environmental sounds. Accuracy for tracking tone and background sounds was particularly low. As conversation length increased, accuracy for all types of information declined.