Web bot protection research: the impact of protection on shopping websites and its relationship to visits and search relevance

Babaritsky Pavel Alexandrovich, Gosudarev Ilya Borisovich

This article describes the study of the interaction of automated systems of information collectors and the methods and means of protection against them opposed to them. Covers working with both automatic collectors, based on the Python programming language, and also discusses examples of various protections that web bots may encounter. Includes data collection results and statistics on a sample of the online shopping segment, such as traffic, relevance and security. Based on the aforementioned data, a forecast was formed about the likelihood of protection, depending on the popularity of the site or its position in the search engine.

Keywords:web-scrapping; web-crawler; security; web-bots, web-parsing.