Home // ACHI 2017, The Tenth International Conference on Advances in Computer-Human Interactions // View article

Sonification of 3D Object Shape for Sensory Substitution: An Empirical Exploration

Authors:
Torkan Gholamalizadeh
Hossein Pourghaemi
Ahmad Mhaish
Gökhan İnce
Damien Jade Duff

Keywords: Sensory substitution; sensory augmentation; point clouds; depth cameras; sound synthesis

Abstract:
An empirical investigation is presented of different approaches to sonification of 3D objects as a part of a sensory substitution system. The system takes 3D point clouds of objects obtained from a depth camera and presents them to a user as spatial audio. Two approaches to shape sonification are presented and their characteristics investigated. The first approach directly encodes the contours belonging to the object in the image as sound waveforms. The second approach categorizes the object according to its 3D surface properties as encapsulated in the rotation invariant Fast Point Feature Histogram (FPFH), and each category is represented by a different synthesized musical instrument. Object identification experiments are done with human users to evaluate the ability of each encoding to transmit object identity to a user. Each of these approaches has their disadvantages. Although the FPFH approach is more invariant to object pose and contains more information about the object, it lacks generality because of the intermediate recognition step. On the other, since contour based approach has no information about depth and curvature of objects, it fails in identifying different objects with similar silhouettes. On the task of distinguishing between 10 different 3D shapes, the FPFH approach produced more accurate responses. However, the fact that it is a direct encoding means that the contour-based approach is more likely to scale up to a wider variety of shapes.

Pages: 18 to 24

Copyright: Copyright (c) IARIA, 2017

Publication date: March 19, 2017

Published in: conference

ISSN: 2308-4138

ISBN: 978-1-61208-538-8

Location: Nice, France

Dates: from March 19, 2017 to March 23, 2017