TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models

Le résumé fourni par la source

Large Multimodal Models (LMMs) have achieved significant success across various tasks.These models usually encode visual inputs into dense token sequences, which are then concatenated with textual tokens and jointly processed by a language model.However, the increased token count substantially raises computational and memory costs during inference.Token pruning has emerged as a promising approach to address this issue.Existing token pruning methods often rely on costly calibration or suboptimal importance metrics, leading to redundant retained tokens.In this paper, we analyze the redundancy differences between visual and textual tokens and propose pruning exclusively on visual tokens.Based on this, we propose a visual token pruning strategy that explicitly preserves both cross-modal alignment and intra-modal informational diversity.We introduce a mutual information-based token pruning strategy that removes visual tokens semantically misaligned with textual tokens, effectively preserving the alignment between the visual and textual modalities.We further refine the retained tokens by maximizing their expected pairwise distances in the latent space to enhance representational quality and reduce redundancy.which is solved efficiently with a greedy algorithm.Extensive experiments demonstrate that our method maintains strong performance while reducing tokens by 88.9% on models such as LLaVA-1.5-7B and LLaVA-NEXT-7B, resulting in a 56.7% improvement in inference speed.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.