Home // ACHI 2018, The Eleventh International Conference on Advances in Computer-Human Interactions // View article

Real-time Shape-based Sensory Substitution for Object Localization and Recognition

Authors:
Hossein Pourghaemi
Torkan Gholamalizadeh
Ahmad Mhaish
Gökhan İnce
Damien Jade Duff

Keywords: sensory substitution; sensory augmentation; point clouds; depth camera; sonification; object tracking

Abstract:
In this paper, we present a new approach to real-time tracking and sonification of 3D object shapes and test the ability of blindfolded participants to learn to locate and recognize objects using our system in a controlled physical environment. In our sonification and sensory substitution system, a depth camera accesses the 3D structure of objects in the form of point clouds and objects are presented to users as spatial audio in real time. We introduce a novel object tracking scheme, which allows the system to be used in the wild, and a method for sonification of objects which encodes the internal 3D contour of objects. We compare the new sonfication method with our previous object-outline based approach. We use an ABA/BAB experimental protocol variant to test the effect of learning during training and testing and to control for order effects with a small group of participants. Results show that our system allows object recognition and localization with short learning time and similar performance between the two sonification methods.

Pages: 45 to 50

Copyright: Copyright (c) IARIA, 2018

Publication date: March 25, 2018

Published in: conference

ISSN: 2308-4138

ISBN: 978-1-61208-616-3

Location: Rome, Italy

Dates: from March 25, 2018 to March 29, 2018