Alignment-Free Multi-Modality Large-Small Model Bidirectional Collaboration with Missing Modality

Le résumé fourni par la source

Different from existing single-modality large-small model collaborations, multi-modality large-small model collaboration is a key but under-explored paradigm where cloud-side multi-modality large model (MM-LM) collaborates with edge-side small models (SMs) to achieve bidirectional task improvements. Nevertheless, this paradigm faces two key challenges. First, effective MM-LM training relies on abundant modality-aligned samples. Nevertheless, due to geographical diversity, parties typically capture non-overlapping data in heterogeneous modalities. In extreme cases, this results in zero overlapping samples across parties, leading to multi-modality alignment absence. Second, the difference in device failure and annotation costs further leads to heterogeneous modality missing ratios among parties. While existing methods leverage MM-LMs for completion, the distribution shift between public pre-training data and private domain data often degrades generation quality. However, addressing this shift via fine-tuning typically demands enough modality-complete data, which is unavailable to parties facing severe missingness, thereby creating domain shift modality completion. To address these challenges, we propose alignment-free multi-modality large-small model bidirectional collaboration with missing modality, named BoMM, which consists of three key components: heterogeneous modality adaptive alignment utilizes instance-level global prototypes to achieve cross-party multi-modality sample alignment, supporting abundant to zero overlapping samples; prototype-anchored preference completion leverages these prototypes to identify the nearest match for missing modalities, supporting arbitrary modality missing with the assumption that only one modality-complete sample per class exists; and quality-aware dynamic sample scheduler dynamically filters generated data in a batch-wise manner, ensuring high-quality samples contribute to fine-tuning. Finally, the experimental results across four multi-modality scenarios clearly demonstrate the superiority of our proposed BoMM.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.