Algorithm for building a website model
Abstract
The analysis of the structure of the website modeling has been carried out. The models of Internet space representation in the form of semantic networks, frame structures and ontology have been analyzed. The web graph model has been chosen to represent the web resource. The pages of a web resource are connected by hyperlinks, which form the internal structure of the resource. To build a model of a website in the form of a web graph, a method and algorithm for scanning the pages of a web resource have been developed. The web resource scanning is performed by in depth searching with the LIFO (Last In - First Out) method. Links are searched by sorting the lines of the page markup text and extracting links by using regular expressions. Only links to pages within the resource are taken into account in the search process, external links are ignored. The crawling procedure is implemented by using the Scrapy framework and the Python. To account for the presence of additional filters used to select pages with criteria, the rules for selecting URL in HTML code have been strengthened. Web resources are scanned to build their web graphs. Storing information by using a list of edges and an adjacency matrix is used in further work with the obtained web graphs. To visualize the obtained graphs and calculate some metric characteristics, the Gephi software environment and the algorithm for stacking the vertices of the Yifan Hu graph has been used. The graph diameters, the average vertex degree, the average path length, the density factor of the graph are used for analysis of the structural connectivity of the graphs studied. The proposed approach can be applied during the site reengineering procedure.
Downloads
References
R. Kumar, P. Raghavan, S. Rajagopalan, D. Sivakumar, A. Tomkins, E. Upfal The Web as a graph. In Proceedings of the 19th Symposium on Principles of Database Systems (PODS). ACM Press. 2000, P. 1–10.
Ling Liu and Özsu Tamer M. Ontology to appear in the Encyclopedia of Database Systems [Электронный ресурс]. Springer-Verlag, 2008. URL: http://tomgruber.org/writing/ontology-in-encyclopedia-of-dbs.pdf (Last accessed: 06.10.2020)
Kaung Myat Htu “Web Ontology Language Analysis (owl) and Semantic Web Technology”, Auditorium, №4 (16), 2017. URL: https://cyberleninka.ru/article/n/analiz-yazyka-veb-ontologii-owl-i-semanticheskaya-veb-tehnologiya (date of access: 06.10.2020) [in Russian]
A.I. Olshevsky “Description of ways to represent web sites in the form of a frame model for the implementation of functional operations in Internet client systems”, Artificial Intelligence, №1, С. 110–116, 2008. [in Russian]
L. Bing Web Data Mining. Springer. 2011, 624 p.
P. Sheu, H. Yu, C. Ramamoorthy, A. Joshi, L. Zadeh “Link Analysis in Web Mining: Techniques and Applications”, Semantic Computing Digital Object Identifier, P. 69–86, 2010.
Lai Wei, Huang Xiaodi “From graph data extraction to graph layout: Web information visualization”, Information Sciences and Interaction Sciences (ICIS), 2010 3rd International Conference on Digital Object Identifier, P. 224–229, 2010.
W. Hall, T. Tiropanis “Web evolution and Web Science”, Computer Networks, Vol. 56, № 18, P. 3859–3865, 2012.
Yu.I. Shokin, A.Yu. Vesnin, A.A. Dobrynin, O.A. Klimenko, E.V. Rychkova “Analysis of the web space of academic communities using webometrics and graph theory”, Information technologists, № 12, С. 31–40, 2014. [in Russian]
N.A. Ermolin, V.V. Mazalov, A.A. Pechnikov “Game-theoretic methods for finding communities in the academic Web”, SPIIRAS Proceedings, 6(55), P. 237–254, 2017. URL: https://doi.org/10.15622/sp.55.10 (date of access: 06.10.2020) [in Russian]
A.L. Gorbunov “Markov Models of Website Traffic”, Internet Mathematics 2007: collection of articles. works of participants in the competition of scientific projects on information retrieval. С. 65–73. 2007
Y. Liu, Z.M. Ma, C. Zhou “Web Markov Skeleton Processes and Their Applications”, Tohoku Mathematical Journal. № 63. P. 665–695. 2011
M. Yadav, N. Goyal “Comparison of Open Source Crawlers- A Review”, International Journal of Scientific & Engineering Research, V. 6, P. 1544-1551, 2015.
T.V. Udapure, R.D. Kale, R.C. Dha “Study of Web Crawler and its Different Types”, Journal of Computer Engineering, Vol. 16(1), P. 3–4, 2014.
M.S. Ahuja, J.S. Bal, Varnica “Web Crawler: Extracting the Web Data”, International Journal of Computer Trends and Technology, №13(3), P. 132–137, 2014.
A.A. Pechnikov, E.M. Sotenko “Crawler programs for collecting data on representative sites of a given subject area - an analytical review”, Modern high technology technologists, № 2, С. 58-62, 2017. [in Russian]
E.M. Pudikova “Review of web crawlers for solving the problem of collecting data on representative sites of a given subject area”, System analysis, Р. 1–16, 2016. [in Russian]
K.V. Gudkov, M.V. Tonkushin “Analysis of automated systems for collecting information on the Internet”, Modern information technologies, № 28, С. 27–31, 2015. [in Russian]
I.S. Blekanov, S.L. Sergeev, I.A. Martynenko “Build subject-oriented web crawlers using a generic kernel”, Scientific and technical statements of the St. Petersburg State Polytechnic University. Informatics. Telecommunications. Control. № 5(157), С. 9–15, 2012. [in Russian]
Y. Hu “Efficient and high quality force-directed graph drawing”, Mathematica Journal, №10, Р. 37–71, 2006.
