RESEARCH AND DEVELOPMENT OF A MECHANISM FOR IMPLEMENTING PRE-CALCULATED TABLES IN THE MERGE DATA LOADING STRATEGY

Zemlin Nikita Alekseyevich, Tyunin Evgeny Borisovich

This article describes the process of loading data from the operational storage layer (ODS) to the user's aggregate and map (DM) area, and focuses on the Merge strategy, which is the most popular and in demand. The use of pre-calculation tables to optimize the data loading process is proposed and the mechanism for implementing this optimization using the Pentaho Data Integration (PDI) tool is discussed. It concludes by stating that the use of pre-calculation tables mechanism for loading data from ODS to DM layer for Merge strategy is a cost-effective solution that allows not to increase the project cost and to continue working with the ever-growing amount of data. Purpose: To develop a mechanism for pre-calculated tables to load data from ODS to DM layer for Merge strategy. Work method or methodology: the paper used methods of designing and creating an information system. Results: the most productive and qualitative method of data loading based on Merge strategy was obtained. Scope of the results: it is advisable to apply the obtained results to large enterprises, which carry out the constant loading of large amounts of data.

Keywords:data loading, ETL, ETL-technologies, information systems, loading strategy, ODS, DM, pre-calculation table