arXiv 2610.23787
DLite: History-free Speculative Decoding for Agentic AI
AI18 upvotes
Authors: Jiaming Xu, Hanzhen Wang, Tianlang Zhao, Jiewen Xiao, Chengze Yuan, Jiayi Pan, Yongkang Zhou, Jiancai Ye +2
Overview
The authors propose DLite, a speculative decoding framework for agentic AI that reuses hidden states from the target model and makes local predictions without maintaining the growing historical context. A lightweight sequential head restores causal dependencies within each block, and a tailored two-stage training strategy trains this architecture.
Introduction
Speculative decoding speeds up LLM inference, but agentic workloads with long, evolving contexts and frequent multi-turn interactions cause existing draft models to lose up to 80% of their acceptance length, while keeping historical KV caches adds substantial system overhead.
Results
DLite achieves a 2-6 acceptance length and outperforms the state-of-the-art method DSpark by up to 70%. These gains hold across general-purpose, long-context, and agentic workloads.
Conclusion
Removing the dependence on ever-growing history makes speculative decoding practical for agentic AI systems.