INTEGRATION OF DATA INTO STORAGE FROM MULTI-FORMAT SOURCE SYSTEMS: METHODS AND PRACTICES

Dementieva Elena Maksimovna

The purpose of our work is to describe the testing of some architectural solutions and methods that ensure reliable, scalable and high-quality data integration into a centralized repository. The relevance of the research is due to the need for effective integration of data from diverse and heterogeneous sources in the context of rapid growth in the volume, speed of receipt and diversity of data, which is a key challenge for corporate and information and analytical systems. The article discusses approaches to combining information from relational databases, APIs, and file storages, followed by uploading it to a ClickHouse-based analytical system. Technologies and tools (Apache NiFi, Apache Airflow, Apache Kafka in conjunction with Debezium, dbt, Great Expectations) used to implement ETL/ELT processes and data quality control are presented. The paper presents the results of modeling various integration scenarios, including batch and streaming downloads. The results of the study confirm the effectiveness of ELT and CDC architectures for building modern data integration platforms capable of providing high performance, fault tolerance and reliability of analytical information.

Keywords:data integration, data warehouse, streaming, ETL/EL, data architecture.