GestoSense: a hand gesture recognition system for individuals with auditory and speech impairments

It is estimated that over 5% of the global population experiences some degree of hearing and speech impairment. However, the scarcity of individuals proficient in sign language has created challenges in ensuring communication accessibility for deaf and hard-of-hearing individuals. In recent years, the exploration of machine learning techniques has contributed to some extent to bridge that gap. This chapter addresses the imperative need for intuitive and inclusive communication for individuals with hearing and speech challenges by creating a fully automated tool, that using the webcam, translates sign gestures into plain English. Focusing on Indian Sign Language (ISL), our work involves first, the creation of a robust imbalanced dataset comprising 80,000 images across 35 classes (Numbers 1–9 and Letters A-Z). The dataset was meticulously curated to capture diverse gestures from various angles and individuals. Then, using the dataset, we trained two convolutional neural network (CNN) models, (1) VGG-16, a pre-trained model, and (2) our custom one. Both models demonstrated proficiency in recognizing a diverse range of hand gestures, achieving F1-Score of 81 and 87 for the VGG-16 and the customized models respectively. The results showcase the superior performance of the custom model over VGG-16, establishing its efficacy for practical applications in real-world scenarios.