Abstract
GestureLinkAI is an innovative approach designed to enhance communication within virtual environments, specifically for American Sign Language (ASL) users. This system utilizes a head-mounted camera to capture the intricate movements of ASL and translates these gestures into spoken words that animate avatars in real-time. It also works in the opposite direction, converting spoken language back into gestures through a cutting-edge gesture-to-speech and speech-to-gesture system.
The technology employs advanced artificial intelligence (AI) models, including adaptations of TimeSformer, CNN-LSTM, and 3D CNN-LSTM, to accurately recognize and interpret ASL gestures in real time. Integrating this gesture recognition system with avatar control in the virtual environment involves a complex data pipeline, enabling seamless communication between the AI models and the game engine. Recognized ASL gestures are translated into fluid avatar animations, allowing users to communicate effortlessly in the virtual world, with avatars that accurately reflect their intended signs.
GestureLinkAI acts as a gateway to inclusive communication, ensuring that every gesture and sign has a voice in the digital realm. By merging advanced AI with immersive environments, this project aims to enhance the virtual experience for ASL users, making every interaction meaningful and every connection authentic.