Sign Language Translator with Deep Learning

Abstract

Currently, there are only 53 qualify sign-language translators offer to around 155 thousand population of deaf-mute in Hong Kong. In 2018, the Hospital Authority received 811 request case for free sign-language translators, yet, there are just 11 translators provide such service. This fully reflected the situation of unbalance demand and supply of translating service for deaf-mute, which cause them to feel helpless to tackle problems in different aspects like court, hospital, etc. To relieve this situation, this project aims to invent a sign-language translator using deep learning. In this paper, it described the mechanism of both multi-layer Convolutional Neural Network (CNN) and two-stacked Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) is adapted to achieve the task. Starting from a raw video, utilize multi-layer CNN in the OpenPose algorithm to detect human skeleton in the video and extract as a raw data input for the LSTM training model to complete sign language translation. The final decision on the training model is selected by comparing among different machine learning frameworks such as CNN, RNN, and RNN-LSTM. This project required to learn with temporal data. For CNN, it is more favorable to process spatial data. For RNN, it had a limitation on learning long-term temporal dependencies. Yet, LSTM is a modified version of RNN with a "memory cell" which had a feedback connection to process either single or a sequence of data. This can solve the problem of vanishing gradient occurs in RNN. In this scenario, LSTM is more suitable to process a sequence of correlate key point features and return with its linguistic meaning. This sign language translator is successfully conducted in this project to translate 5 different types of sign languages. They are 'bleed', 'fever', 'pregnant', 'pain' and 'vomit'. Those words are selected as it is commonly used in the hospital. It is believed that further development of the current model with a wide range of sign language detected in real-time is possible to relieve the situation of insufficient supply of sign language translator.