Matrix-Structured Hierarchical Convolutional Modeling for Pronunciation Assessment and Mispronunciation Detection

Le résumé fourni par la source

We introduce M3C1, a hierarchical, matrix-structured convolutional framework for automatic pronunciation assessment (APA) and phone-level mispronunciation detection (MDD). Heterogeneous representations–GoP (LPP/LPR), canonical-phone embeddings, self-supervised learned representations (HuBERT, Wav2Vec 2.0, WavLM), and prosodic features–are reorganized into column-aligned matrices and compressed by a self-designed CNN block (Compact Convolutional Condenser) across feature extraction, phone, word, and utterance levels. Multi-aspect attention is used to correlate same-level aspects, and APA is jointly trained with an auxiliary phone classifier for MDD. In a GoP-only parity setting, M3C outperforms GOPT and recent CNN-Based models on the speechocean762 corpus, with the largest gains at word level (+19.4% and +7.2% respectively); with all the features, our model also improves state of the art at word level and MDD F1 (+15%). Ablation studies confirm the importance of the proposed matrix structured feature extraction and explicit triphone context, highlighting the effectiveness of proximity-driven local modeling for competitive APA.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.