Multi-Scale Adaptive Distillation

Le résumé fourni par la source

Knowledge Distillation (KD) is used to transfer knowledge from a large pre-trained teacher network to a lightweight student network, facilitating efficient deployment on resource-constrained hardware. However, existing methods overlook a key issue: the imbalance between feature scale and discriminative capability. In this paper, we propose a new method, Multi-scale Adaptive Distillation (MAD), which addresses this imbalance by considering features at different scales and dynamically adjusting distillation weights. MAD enables student learning across multiple scales, enhancing its ability to learn key details of various sizes through a local adaptive weight module; simultaneously, it captures the overall structural relationships between different scales by introducing a cross-scale logit correlation matrix, ensuring that the student model can comprehensively learn semantic features. Experimental results on public datasets demonstrate that our method effectively trains reliable student networks under various network architectures.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.