Home // ICAS 2015, The Eleventh International Conference on Autonomic and Autonomous Systems // View article

Self-organized Architecture for Sharing Data Streams at Large Scale

Authors:
Nicolas Hidalgo
Erika Rosas

Keywords: Stream Processing; Peer-to-Peer networks; Large Scale Computing; Publish/Subscribe.

Abstract:
Stream Processing Engines are designed to deal with real-time computing of massive data streams generated on social networks, news feeding, satellite images, sensor devices, among other sources. For example, in the context of the Internet of Things and Smart Cities, a high volume of data it is expected to be distributed geographically. In this context, the re-use of processed stream enables resource optimization by avoiding re-computation, enabling to provide aggregation and global data visualization. We propose a self-organized architecture to share data streams, which enables resource localization over a scalable, fault-tolerant Distributed Hash Table structure. The Stream Processing Engines are organized into a structured peer-to-peer network and they exploit a Publish/Subscribe system to publish and locate preprocessed streams, possibly in other geographic regions. In order to deal with communication latency problems in the peer-to-peer network, we propose a latency-aware algorithm that estimates distance between the nodes in the system.

Pages: 57 to 62

Copyright: Copyright (c) IARIA, 2015

Publication date: May 24, 2015

Published in: conference

ISSN: 2308-3913

ISBN: 978-1-61208-405-3

Location: Rome, Italy

Dates: from May 24, 2015 to May 29, 2015