Tsinghua university unveils open-source AKV system for enhancing LLM agent efficiency

Tsinghua University and the University of Hong Kong have open-sourced AKV, a novel serving system designed to optimize KV cache engineering for large language model (LLM) agents. The system allows agents to selectively drop and reposition KV cache entries, reducing computational overhead while preserving critical information. Across three models and four benchmarks, AKV achieved a 27.7% reduction in average cost and a 1.7-point increase in task scores. The code, licensed under Apache-2.0, is built on SGLang. AKV introduces three cache operations—append, drop, and repos—enabling efficient reuse of retained entries without recomputation. By avoiding the need for re-prefilling, the system significantly improves performance, particularly in long-context tasks. On the BrowseComp-Plus benchmark, GPT-OSS-120B achieved a 37% pass rate with AKV, compared to 24% with the standard approach. The system also supports stateless, OpenAI-compatible interfaces, allowing clients to specify which messages to drop or reposition. Serving tests demonstrated AKV’s efficiency, with 1.18x output and 1.53x input throughput compared to other methods. On 100 SWE-bench Verified tasks, AKV achieved a 47% pass rate, outperforming the standard approach’s 40.7%. The project’s GitHub repository and paper are now publicly available for further exploration.