DEVELOPMENT AND RESEARCH OF A DATA DEDUPLICATION SERVICE FOR STORAGE SYSTEMS

Pantykhin Andrey Maksimovich, Gladun Vladimir Vadimovich, Malinin Ilya Igorevich, Molodyakov Sergey Aleksandrovich

This paper examines a deduplication service based on hash functions to minimize the volume of stored data. The main algorithm involves segmenting data into fixed-size blocks, calculating unique hash values for each segment, and saving only unique data blocks while creating references for duplicates. The technological stack includes Python, MongoDB, and the MongoEngine library. The paper presents research results related to the use of different hashing algorithms and data segment sizes.

Keywords:data deduplication, hash functions, data storage systems, storage optimization, MongoDB, Python, MongoEngine