Modeling an opportunistic ETL agent based system using YAWL

Abstract

ETL systems are responsible for populating data warehouses' storage structures with historical subject-oriented business data. To do that in the most appropriated manner ETL systems require special computational means, involving frequently the manipulation of large amounts of data that come usually from disparate information sources. Planning and designing ETL systems are very complex tasks, where the occurrence of errors is not rare. In some particular situations, this may jeopardize the successful implementation of the entire system. Therefore, ETL systems modeling is a very important activity in an ETL project, since it allows for sketching and validating different kinds of implementations accordingly all operational requirements defined. This will reduce significantly project costs originated by misinterpreted or badly mapped specifications. In this work we propose a different approach for modeling ETL systems, using ETL patterns as main systems components of an ETL system that will be used and tested in a standard simulation environment using the workflow language YAWL. The system is supported by a cooperative community of agents that are in charge to put in practice the tasks specified of a specific ETL pattern and, together, execute the entire set of tasks required by a particular ETL system package.