Home // International Journal On Advances in Software, volume 7, numbers 1 and 2, 2014 // View article

The Data Checking Engine: Complex Rules for Data Quality Monitoring

Authors:
Felix Heine
Carsten Kleiner
Arne Koschel
Jörg Westermayer

Keywords: Data Quality, Quality Rules, Data Analysis, Data Quality Monitoring, Data Warehouses

Abstract:
In the context of data warehousing and business intelligence, data quality is of utmost importance. However, many mid-size data warehouse (DWH) projects do not implement a proper data quality process due to huge up-front investments. Nevertheless, assessing and monitoring data quality is necessary to establish confidence in the DWH data. In this paper, we describe a data quality monitoring system: The “Data Checking Engine” (DCE). The goal of the system is to provide DWH projects with an easy and quickly deployable solution to as- sess data quality while still providing highest flexibility in the definition of the assessment rules. It allows to express complex quality rules and implements a two-staged template mechanism to facilitate the deployment of large numbers of similar rules. While the rules themselves are SQL statements the tool guides the data quality manager through the process of creating rule templates and rules so that it is rather easy for him to create large sets of quality rules. The rule definition language is illustrated in this paper and we also demonstrate the very flexible capabilities of the DCE by presenting examples of advanced data quality rules and how they can be implemented in the DCE. The usefulness of the DCE has been proven in practical implementations at different clients of SHS Viveon. An impression of the actual implementations of the system is given in terms of the system architecture and GUI screenshots in this paper.

Pages: 171 to 181

Copyright: Copyright (c) to authors, 2014. Used with permission.

Publication date: June 30, 2014

Published in: journal

ISSN: 1942-2628