An ETL pattern for log configuration and analysis

Abstract

In many scenarios, such as the ones related to Data Warehousing Extract-Transform-Load (ETL) processes, logging techniques are usually applied for capturing event metrics across system levels for system auditing and system recovery. The diversity of strategies and architectures of the toolset used to support the ETL implementation introduces another layer of complexity, both for system development and audit. Although a valuable system diagnosis resource for the development team, logging is generally underestimated, being used only when the system reveals unexpected behaviours and not to drive the ETL system evolution. We believe that the use of logs for steering ETL development and maintenance can improve significantly global system quality. However, this approach is only effective if flexible and efficient logging systems exist. In this paper, we describe a Log Pattern used in a pattern-oriented approach for ETL systems development, which provides a configurable and flexible component for using to drive ETL development and maintenance phases.