BUILDING A DISTRIBUTED WEB APPLICATION NETWORK SCANNING SYSTEM WITH SPIDERING AND CROWLING FUNCTIONS

Shipulin Georgy Farizovich

The article is devoted to the consideration of issues related to the collection of information about the functioning of web applications in the framework of security assessment and penetration testing. The paper considered the main elements of the information collection process during penetration testing according to the OWASP Web Security Testing Guide web application penetration testing methodology, including the categories of data collected and aspects of their analysis for metadata, information about the components of the web application and the technology stack used. Based on the analysis of existing automated web scanning tools for web applications, their main limitations were identified (insufficient scanning coverage and completeness of data collection, mono-tasking). The architecture of the developed distributed web application network scanning system with component-level spidering and crowling functions is described, and the results of its experimental launches are presented.

Keywords:penetration testing, OWASP, web application, network scanning, crawling, spidering, information collection, information security