Paper Title:
Design of the Distributed Web Crawler
  Abstract

On the current scale of the Internet, the single web crawler is unable to visit the entire web in an effective time-frame. So, we develop a distributed web crawler system to deal with it. In our distribution design, we mainly consider two facets of parallel. One is the multi-thread in the internal nodes; the other is distributed parallel among the nodes. We focus on the distribution and parallel between nodes. We address two issues of the distributed web crawler which include the crawl strategy and dynamic configuration. The results of experiment show that the hash function based on the web site achieves the goal of the distributed web crawler. At the same time, we pursue the load balance of the system, we also should reduce the communication and management spending as much as possible.

  Info
Periodical
Advanced Materials Research (Volumes 204-210)
Edited by
Helen Zhang, Gang Shen and David Jin
Pages
1454-1458
DOI
10.4028/www.scientific.net/AMR.204-210.1454
Citation
X. Chen, W. J. Li, T. J. Zhao, X. H. Piao, "Design of the Distributed Web Crawler", Advanced Materials Research, Vols. 204-210, pp. 1454-1458, 2011
Online since
February 2011
Export
Price
$32.00
Share

In order to see related information, you need to Login.

In order to see related information, you need to Login.

Authors: Xiao Yan Xiong, Miao Zhang, Xiao Ping Li, Shao Juan Yu
Abstract:Based on chaotic characteristics in vertical direction of vibrating screen sides, nonlinear methods were proposed to diagnose crack of...
1258
Authors: Ying Lin Li, Li Hui Cao, Lian He Yang
Abstract:Weft knitted pattern design is one of the most important compositions of textile CAD. Traditional pattern design has a higher request on...
576
Authors: Yong Hua Zhang, Jian Hui He, Guo Qing Zhang
Abstract:This paper aims to understand influence of the obliquity of fin ray on its motion performance. An environment-friendly propulsion system...
267
Authors: Ioana Pintilie, Francesco Moscatelli, Roberta Nipoti, Antonella Poggi, Sandro Solmi, Lars S. Løvlie, Bengt G. Svensson
Abstract:The effect of nitrogen (N) introduced by ion implantation at the SiO2/4H-SiC interface on the capacitance of the MOS capacitors is...
326
Authors: Yi Mei, Fang Ping Wang, Qiao Ying Liu, Yu Tao Mao
Abstract:To solve the thermal deformation caused by thermal load of heavy machinery gearbox, it is established that coupled analysis model to carry...
651