Home // ALLDATA 2021, The Seventh International Conference on Big Data, Small Data, Linked Data and Open Data // View article

Distributed Search on a Large Amount of Log Data

Authors:
Fabrice Mourlin
Charif Mahmoudi
Guy Lahlou Djiken

Keywords: log; indexing; Big Data; streaming; planning maintenance.

Abstract:
Log analysis is the basis for planning maintenance operations. It is used to predict software and hardware failures. Their costs can be high, ranging from activity blocking to data loss, even penalties for late results. In this context, to achieve an even better quality of service, we have built a distributed application for collecting and analyzing software logs. Key properties had to be taken into account such as the number of logs per hour, the variety of formats of these logs or the indexing of information known in these logs. Our approach responds to these properties by using a Big Data cluster and installing a custom indexing engine for software log analysis. Our results show a reduction in software failures and, therefore, better availability of software services under monitoring. This result leads to rethinking software maintenance and reviewing the sizing of our cluster according to the number of monitored applications correlated with the throughput of each one.

Pages: 30 to 37

Copyright: Copyright (c) IARIA, 2021

Publication date: April 18, 2021

Published in: conference

ISSN: 2519-8386

ISBN: 978-1-61208-842-6

Location: Porto, Portugal

Dates: from April 18, 2021 to April 22, 2021