MambaGesture2: Co-Speech Gesture Generation via Hierarchical Fusion and Spatiotemporal Aggregation

Le résumé fourni par la source

Co-speech gesture generation plays a vital role in producing synchronized and natural human gestures, thereby enhancing the realism of avatars in virtual environments. Although diffusion models have shown strong generative capabilities, their combination with transformer-based architectures often incurs high computational costs due to the quadratic complexity of self-attention. Moreover, as a temporal sequence modeling task, existing methods frequently struggle to effectively capture multi-scale temporal dynamics inherent in speech and gesture signals. To address these challenges, we propose MambaGesture2, a novel framework that integrates a Mamba-based denoising network, Hierarchical U-Net Gesture Mamba (HUG-Mamba), with a multimodal feature fusion module, SEAD. HUG-Mamba combines the efficient state-space modeling of Mamba blocks with the hierarchical sampling of the U-Net architecture, significantly improving temporal coherence and computational efficiency. We further introduce the Temporal-Stratified Fusion (TSF) module to capture diverse temporal scales via multi-scale learning, and the Spatial-Temporal Cascaded Aggregation (STCA) module to enhance spatial-temporal feature aggregation. Extensive experiments on the multi-modal BEAT2 and SHOW datasets demonstrate that our approach achieves state-of-the-art performance across multiple quantitative metrics, while substantially reducing model complexity and inference time. The results validate the effectiveness of our architectural innovations in generating diverse, realistic, and temporally consistent co-speech gestures. Project page:https://fcchit.github.io/mambagesture2.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.