Home // SOFTENG 2017, The Third International Conference on Advances and Trends in Software Engineering // View article

Chimera: A Distributed High-throughput Low-latency Data Processing and Streaming System

Authors:
Pascal Lau
Paolo Maresca

Keywords: Distributed computing, High performance computing, Data systems

Abstract:
On a daily basis, Internet services experience growing amount of traffic that needs to be ingested first, and processed subsequently. Technologies to streamline data target horizontal distribution as design tenet, giving off maintainability and operational friendliness. The advent of the Internet of Things (IoT) and the progressive adoption of IPv6 require a new generation of data streamline platforms, bearing in mind easy distribution, maintainability and deployment. Chimera is an ultra-fast and scalable Extract Transform and Load (ETL) platform, designed for distribution on commodity hardware, and to serve ultra-high volumes of inbound data while processing in real-time. It strives at putting together top performance technologies to solve the problem of ingesting huge amount of data delivered by geographically distributed agents. It has been conceived to propose a novel paradigm of distribution, leveraging a shared nothing architecture, easy to elastically scale and to maintain. It reliably ingests and processes huge volumes of data: operating at the line rate, it is able to distribute the processing among stateless processors, which can join and leave the infrastructure at any time. Experimental tests show relevant outcomes intended as the ability to systematically saturate the I/O (network and disk), preserving reliable computations (at-least-once delivery policy).

Pages: 16 to 24

Copyright: Copyright (c) IARIA, 2017

Publication date: April 23, 2017

Published in: conference

ISSN: 2519-8394

ISBN: 978-1-61208-553-1

Location: Venice, Italy

Dates: from April 23, 2017 to April 27, 2017