SuperSpec: Enhanced Verification and Sampling for End-to-End LLM Speculative Decoding

Le résumé fourni par la source

Modern LLM decoding has the drawbacks of high cost and slow speed, and speculative decoding has been shown to be an effective solution to this problem. However, the inference latency still poses a significant challenge to maintaining service level objectives (SLOs) in systems that employ multiple draft models for speculative decoding. The verification phase in such systems if reliant on tree attention often constitutes a bottleneck especially when draft sequences lack common prefixes and substantially underutilizes GPU parallelism while increasing end-to-end latency. We introduce SuperSpec, an end-to-end speculative decoding system designed to co-optimize verification, sampling and draft generation. SuperSpec integrates three pivotal innovations: an Efficient Batch Verifier, which substitutes treebased flattening with batch parallel validation and layer-wise KV Cache replication; a Global Optimal Sampler, which assesses all candidate sequences within a batch to ascertain the longest valid path, thereby circumventing the local optima frequently encountered in tree-based rejection sampling; and a Dynamic Adaptive Multi-Drafter, which dynamically modulates the speculative length (K) for each drafter predicated on real-time idleness metrics and acceptance rates. Empirical evaluations of Qwen2.5-72B and the OPT-66B on various datasets show that SuperSpec improves average acceptance rate by 6.4% to 30.2%, and the end-to-end inference acceleration ratio by 7.12% to 62.06%, when compared to the state-of-the-art tree-based speculative decoding system SpecInfer. These improvements were achieved without compromising the quality of text generation, making SuperSpec an effective solution for accelerating LLM inference.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.