Home // International Journal On Advances in Intelligent Systems, volume 15, numbers 1 and 2, 2022 // View article

Automatic Recognition of Continuous Sign Language for Public Services

Authors:
Robson Silva de Souza
José Mario De Martino
Janice Gonçalves Temoteo Marques
Ivani Rodrigues Silva

Keywords: Brazilian Sign Language; Sign language recognition; Continuous signing; long short term memory; connectionist temporal classification

Abstract:
In this article, we present an automatic image recognition approach for assisting the communication between deaf people and hearing physicians. The aim of the approach is to help the interaction and exchange of information during medical interviews and in different public services, such as police departments, hospitals, and citizen service centers. Its scope is the automatic recognition of the continuous signing through the analysis of traditional video and depth data (RGB-D data). Recognition is performed by a cascade of two neural networks. First, a convolutional neural network encodes the visual input and extracts relevant features. Second, a recurrent neural network learns the mapping of the extracted features and transforms them into words. We use the Connectionist Temporal Classification approach to train the recurrent network with videos of different lengths and word sequences. Experiments on two continuous sign language datasets show the effectiveness of our approach, achieving an accuracy of around 91% in the Brazilian Sign Language (Libras) dataset and 94% in Greek Sign Language (GSL) in signer-independent continuous sign language setup.

Pages: 71 to 82

Copyright: Copyright (c) to authors, 2022. Used with permission.

Publication date: June 30, 2022

Published in: journal

ISSN: 1942-2679