Multimodal Brain-Computer Interfaces Using Cross-Modal Transformer Fusion Framework

Le résumé fourni par la source

Recently, the Brain-Computer Interfaces (BCIs) are emerging as an important tool for neurorehabilitation, cognitive monitoring and human-machine interaction where the multimodal integration of neural signals improves system robustness. Among these available modalities, the Electroencephalography (EEG) provides high temporal resolution while functional Near-Infrared Spectroscopy (fNIRS) captures complementary hemodynamic responses. However, the existing methods often depends on decision-level fusion which fails to fully exploit the deep cross-modal relationships between EEG and fNIRS. Hence, this research proposes Crossattention Transformer-based Fusion Framework (CTFF) for more effective multimodal representation. Initially, the input data is collected from EEG-fNIRS dataset which contains synchronized multimodal brain data. Then, the collected data is preprocessed with Independent Component Analysis (ICA) and Band-Pass Filter (BPF) to remove non-neural artifacts and isolate task-related signals respectively. Further, the modalityspecific encoders are trained with contrastive pretraining to learn robust latent embeddings and align modalities in a shared space. After that, an intermediate fusion stage employs a crossattention transformer to capture temporal dependencies and nonlinear interactions across modalities. Finally, the pooled fused features are classified through a Deep Neural Network (DNN) with auxiliary losses for generalization and uncertainty estimation. The proposed CTFF achieved better results in terms of accuracy (99.89 %) than existing Schizophrenia Diagnosis Framework (Schizo-Net) model.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.