SIGN-AIR: real-time sign language recognition and translation using gesture-aware neural networks on aerial robotic platforms

Résumé fourni par la source

Abstract Communication barriers experienced by deaf and hard-of-hearing individuals remain a significant challenge, particularly in situations where conventional communication infrastructure is unavailable or fixed-camera systems have limited coverage. This paper presents SIGN-AIR, a UAV-assisted framework for real-time sign language recognition and bidirectional translation. Unlike conventional approaches based on fixed cameras, wearable sensors, or depth cameras, the proposed framework integrates a drone-mounted stabilised RGB camera with onboard edge computing to enable flexible gesture recognition in indoor and outdoor environments. The system combines MediaPipe-based hand landmark extraction with a hybrid CNN–BiLSTM architecture to model both the spatial and temporal characteristics of American Sign Language gestures. Experimental evaluation on a combined dataset comprising the ASL Alphabet and WLASL achieved an overall recognition accuracy of 98.63%, together with high precision, high recall, and a median inference latency of less than 2 s. These results demonstrate the technical feasibility of the proposed UAV-assisted recognition framework under the evaluated experimental conditions and indicate its potential to support communication in scenarios where fixed-camera systems may be affected by occlusions, limited fields of view, or environmental variability. Furthermore, the proposed bidirectional translation pipeline has the potential to facilitate both sign-to-speech and speech-to-sign communication for prospective applications in accessibility, education, public service interactions, and emergency response. Although the current study demonstrates promising technical performance, comprehensive real-world field validation under representative operational conditions remains an important direction for future work.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.