Home // International Journal On Advances in Life Sciences, volume 13, numbers 1 and 2, 2021 // View article
Authors:
Vasileios Kassiano
Anastasios S. Kesidis
Thanos G. Stavropoulos
Spiros Nikolopoulos
Ioannis Kompatsiaris
Keywords: ontology; rules; natural language; SPIN; situational awareness; spatial awareness; localization; deafblindness.
Abstract:
Deafblindness is a debilitating condition that hinders communication, awareness of the surroundings and ultimately independent living. Technological advancements in image analysis and haptics promise to support patients. Yet, machine-interpretable representations, interpretation and feedback to bridge the gap between perceiving the environment and communicating this to the user, are lacking. This paper presents a rule-based framework that supports object localization, spatial and situational awareness and natural language feedback for the deafblind. The framework utilizes ontologies as an interoperable data model to represent knowledge about the environment and rules to extract object localization, spatial and situational awareness as well as to form appropriate natural language responses. The rule set is expressed in the SPARQL Inferencing Notation (SPIN), which enables simplicity and flexibility in rule definition. A dashboard web application visualizes the streaming detections perceived from the environment and allows real-time queries and responses. A proof-of-concept scenario is realized with synthetic data from a realistic environment, showing use cases in detecting surroundings, locating an object and being aware of a situation (e.g., people wearing a facemask as a COVID-19 precautionary measure). In the future, the platform will support connection to a data exchange message bus to receive detections from image analysis and communicate actions towards haptic hardware to enable testing by patients.
Pages: 42 to 53
Copyright: Copyright (c) to authors, 2021. Used with permission.
Publication date: December 31, 2021
Published in: journal
ISSN: 1942-2660