Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning

Le résumé fourni par la source

Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to achieve a favorable accuracy-efficiency trade-off. It is equipped with a new Top-Down Compression paradigm that strategically compresses visual tokens while preserving a substantial amount of critical visual information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance using both the Visual and Native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75%-98% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.