Home // HEALTHINFO 2021, The Sixth International Conference on Informatics and Assistive Technologies for Health-Care, Medical Support and Wellbeing // View article

Automatic Recognition of Continuous Signing of Brazilian Sign Language for Medical Interview

Authors:
Robson Silva de Souza
José Mario De Martino
Janice Gonçalves Temoteo Marques
Ivani Rodrigues Silva

Keywords: Libras, Sign language recognition, Continuous signing, long short term memory, connectionist temporal classification

Abstract:
In this article, we present an automatic image recognition approach for assisting the communication between deaf patients, speakers of the Brazilian Sign Language (Libras), and hearing physicians. The aim of the approach is to help the interaction and exchange of information during medical interviews. Its scope is the automatic recognition of the continuous signing of Libras through the analysis of traditional video and depth data (RGB-D data). Recognition is performed by a cascade of two neural networks. The first, a convolutional neural network, encodes the visual input and extracts relevant features. The second, a recurrent neural network, learns the mapping of the extracted features into Brazilian Portuguese words. To train the recurrent network with videos of different lengths and word sequences, we use the Connectionist Temporal Classification approach. Experiments using a dataset of 280 videos encompassing 56 sentences composed of 67 different signs results in an accuracy of round 91%.

Pages: 41 to 46

Copyright: Copyright (c) IARIA, 2021

Publication date: October 3, 2021

Published in: conference

ISSN: 2519-8491

ISBN: 978-1-61208-916-4

Location: Barcelona, Spain

Dates: from October 3, 2021 to October 7, 2021