Tsinghua and CUHK researchers open source AKV cache server for agents

Researchers from Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, a key-value cache engineering serving system for large language model agents. MarkTechPost reported that the software is built on SGLang and released under an Apache-2.0 open-source license. [1]

Instead of summarizing prior conversation history, the system allows artificial intelligence agents to drop selected key-value cache entries and reposition the remaining cache. MarkTechPost reported that context engineering techniques typically invalidate the full cached prefix whenever text history is edited. [1]

Across evaluations using three language models and four agent benchmarks, the AKV system reduced average inference costs by 27.7 percent. MarkTechPost also reported that average task performance scores increased by 1.7 points across the tested benchmarks. [1]

Web caching optimizes system latency by storing data for future requests, a concept computer scientists Jim Gray and Prashant Shenoy analyzed for web page reuse in 2000. Open-source software licenses like Apache-2.0 allow public access to artificial intelligence frameworks and platforms. [2][3]