A systematic comparative analysis of knowledge distillation paradigms across a family of efficient hybrid 3D CNN architectures

Le résumé fourni par la source

Computational demands remain a major obstacle for deploying vision-based Human Action Recognition models in resource-constrained settings. Although state-of-the-art 3D CNNs deliver high performance, their inference time can limit their use in real-time applications. To address this, we designed six lightweight models similar in depth to the 3D ResNet frameworks such as R3D and R (2+1) D, but varying in their channel-width compression and the 3D kernel representations adopted across their layers. As reducing their sizes is associated with limitations on their learning capabilities, we adopted Knowledge Distillation (KD) to improve their performance. Based on that, we considered the R (2+1) D framework as the expert Teacher to train our models through four KD paradigms: response, feature, attention, and decoupled. Our findings show that all KD methods improved baseline performance, despite different accuracy gains that were heavily influenced by the Students’ architecture. Furthermore, we investigated the behavior of these six variants by analyzing their class-level performance when trained by each KD type. The results show that feature-based KD is the most effective strategy among the four KD paradigms. Our experiments also justify the importance of internal layer architecture and its alignment with the Teacher architecture in receiving more knowledge. We evaluated our best lightweight model on UCF101 and HMDB51, and recorded an accuracy of 90.83% vs. 94.74% for the Teacher on UCF101. The results indicated a retained 95% of the performance, while operating at 74% fewer parameters and running 2.9 times faster than the Teacher in our measured tests.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.