Leveraging LLMs, computer vision, and ChatGPT to ease communication between deaf and hearing individuals

Abstract

Hearing and hard of hearing individuals might struggle to communicate without the help of a human interpreter. However, machine learning is a tool that can help enable ease of communication between both groups of people. This project involves an application that uses a convolutional neural network to solve this problem. This convolutional neural network model was custom-made from 46,000 augmented training images and tested with 8000 images. Its purpose in the application is to take camera frames of ASL fingerspelling as input and then translate it into text. This alphabet is appended to a sentence variable where it is stored. Once the user has completed a sentence, a ChatGPT API will correct any grammatical errors or misclassifications that the model made. With this ChatGPT edited sentence, a Play.HT API can turn it to speech in order to further smoothen the language barrier. A background cropping feature was added as well, which takes frames from a camera and uses the MediaPipe module to locate the hand, after which it will remove any part of the image except the hand in order to reduce noise for the model. The final model achieved a 97.5% accuracy and 0.1% loss with the testing dataset and also had a strong confusion matrix as well, with only some mistakes with similar looking characters such as M and N. The model also worked better in different rooms since the background noise was removed from the model entirely. Overall, this application is an innovative step for using AI and machine learning to slowly removing the language barrier between hard of hearing and hearing individuals.