Cross-modal audio-text attention for multimodal multitask speech emotion recognition in low-resource Urdu

Résumé fourni par la source

Speech Emotion Recognition (SER) in low-resource languages remains challenging due to limited annotated data, speaker variability, and the multimodal nature of emotional expression. This paper repositions established components wav2vec 2.0, XLM-R, cross-modal attention, and multitask affective modeling into a framework jointly validated across speaker-independent, cross-lingual zero-shot, and attribution-faithfulness generalization for Urdu, a combination not jointly reported in prior Urdu SER work. The proposed multimodal multitask model achieves 91.3% emotion recognition accuracy on the Urdu Speech Emotion Corpus (UrSEC), outperforming strong audio-only and text-only baselines, with joint valence-arousal learning consistently improving over emotion-only training. Speaker-independent evaluation shows a performance drop relative to random-split testing but confirms substantial robustness to speaker-specific bias. Cross-lingual zero-shot evaluation on English datasets yields 80.2 ± 1.3% (IEMOCAP) to 86.3 ± 0.9% (CREMA-D) accuracy without fine-tuning, indicating substantial cross-lingual transfer, though this alone does not establish full language-neutrality. All performance gains are statistically validated across multiple runs, and attention/Integrated Gradients analyses, supported by quantitative faithfulness testing, show the model relies on emotionally salient acoustic regions and Urdu tokens rather than spurious correlations.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.