Home // International Journal On Advances in Life Sciences, volume 14, numbers 3 and 4, 2022 // View article

Sensor Glove Approach for Continuous Recognition of Japanese Fingerspelling in Daily Life

Authors:
Yuhki Shiraishi
Akihisa Shitara
Fumio Yoneyama
Nobuko Kato

Keywords: Sign language; Japanese fingerspelling; Sensor glove; Recognition; Convolutional neural network; Long short-term memory.

Abstract:
To achieve smooth communication between the deaf and hard of hearing and hearing people, we developed a Japanese fingerspelling (JF) recognition system based on sensor gloves. A light and inexpensive sensor glove was adapted for the daily use of the system. We conducted evaluation experiments using a convolutional neural network (CNN) to recognize 76 characters in JF. The target JF alphabet included 35 characters for dynamic fingerspelling, and required both finger and wrist movement. The experimental results show that the average recognition rate of the developed system was approximately 70.0%. Additionally, we conducted a continuous fingerspelling recognition experiment using CNNs and long short-term memory (LSTMs) networks, aiming to recognize consecutive fingerspelling. We proposed a dataset to exploit the characteristics of JF and selected 64 words according to the finger flexion, direction, and movement differences among various signers. Using the collected data, we then conducted evaluation experiments with seven types of neural networks. The overlapping characteristics present in JF were exploited because finger flexion, finger extension, hand direction, and hand movements vary significantly among people currently learning sign language, people corresponding in Japanese sign language (JSL), and people using JSL in their daily lives. Consequently, the average recognition rate (micro F-measure) of 76 JF characters was approximately 92.1%. Based on the results of single fingerspelling and continuous fingerspelling recognition experiments, we discussed the issues concerning the recognition of JF characters and development of sign language recognition systems.

Pages: 53 to 70

Copyright: Copyright (c) to authors, 2022. Used with permission.

Publication date: December 31, 2022

Published in: journal

ISSN: 1942-2660