Enhancing Emotional Congruence in Sensory Substitution

Le résumé fourni par la source

Sensory substitution devices (SSDs) enable the compensation of sensory loss by translating information from one modality to another. However, current SSDs face limitations in conveying emotional content, which reduces user engagement and acceptance. This study investigates emotional coherence in visual-to-auditory sensory substitution, focusing on the mapping between visual stimuli and musical features. We introduce a novel experimental protocol that systematically measures emotional responses to various combinations of visual and auditory stimuli monitoring facial expression and physiological parameters, and using the Self-Assessment Manikin (SAM) scale. The collected database, containing data from 36 participants, is analysed to answer three research questions: 1) explore the role of valence and arousal dimensions in driving associations between visual content and musical characteristics, 2) verify emotional coherence between the SAM and the collected data, and 3) predict appropriate musical features for visual stimuli based on emotional response data. Results from our analyses revealed a strong preference for valence-matched stimuli over arousal-matched alternatives, with participants selecting valence-matched options in 83.0% of cases, compared to only 6.0% for arousal-matched stimuli. Furthermore, classical machine learning algorithms were successfully employed to classify high and low scores of the SAM metrics using the multimodal behavioural data, achieving 75.4% accuracy for dominance and 63.4% for arousal. Finally, the developed predictive models show promising results in predicting the selected song, with 67.9% in accuracy, and yield exceptionally low error rates in predicting musical characteristics, with mean squared errors reaching approximately$10^{-29}$for some musical features. These results provide a foundation for developing emotion-aware sensory substitution systems that maintain emotional congruence in the translation from visual to auditory modalities, potentially enhancing user engagement and acceptance of SSDs.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.