Web Page Data Collection Based on Multithread

Wen Tao Liu

doi:10.4028/www.scientific.net/AMM.347-350.2575

Paper Titles

Reflection of the Nation Cybersecurity's Evolution
p.2553

A Semantic Query Method for Deep Web
p.2559

The Petri Net Model of the Requirements and Service Composition
p.2564

The Design and Implementation of Florists Shop Online System Based on J2EE
p.2571

Web Page Data Collection Based on Multithread
p.2575

Functional Link Artificial Neural Networks Filter for Gaussian Noise
p.2580

Eigenvalue Decomposition Based Modified Newton Algorithm
p.2586

Bayesian Networks Application in Multi-State System Reliability Analysis
p.2590

Noether Theorem of a Kind of Singular Integral Equation with Hilbert Kernel on Closed Contours
p.2596

HomeApplied Mechanics and MaterialsApplied Mechanics and Materials Vols. 347-350Web Page Data Collection Based on Multithread

Web Page Data Collection Based on Multithread

Abstract:

The web data collection is the process of collecting the semi-structured, large-scale and redundant data which include web content, web structure and web usage in the web by the crawler and it is often used for the information extraction, information retrieval, search engine and web data mining. In this paper, the web data collection principle is introduced and some related topics are discussed such as page download, coding problem, updated strategy, static and dynamic page. The multithread technology is described and multithread mode for the web data collection is proposed. The web data collection with multithread can get better resource utilization, better average response time and better performance.

You might also be interested in these eBooks

View Preview

Info:

Periodical:

Applied Mechanics and Materials (Volumes 347-350)

Pages:

2575-2579

DOI:

https://doi.org/10.4028/www.scientific.net/AMM.347-350.2575

Citation:

Cite this paper

Online since:

August 2013

Authors:

Wen Tao Liu

Keywords:

Data Collection, Multithread, Web Page

Export:

RIS, BibTeX

Price:

Permissions CCC:

Request Permissions

Permissions PLS:

Request Permissions

Сopyright:

Citation:

References

[1] Christopher Olston and Marc Najork, Web Crawling, Foundations and Trends in Information Retrieval, (2010).

DOI: 10.1561/1500000017

Google Scholar

[2] Carlos Castillo and Ricardo Baeza-Yates. A new crawling model. In Poster proceedings ofthe eleventh conference on World Wide Web, Honolulu, Hawaii, USA, May (2002).

Google Scholar

[3] Soderland, S., Learning information extraction rules for semi-structured and free text. Journal of Machine Learning, 34(1-3): 233-272, (1999).

Google Scholar

[4] Freitag, D., Information extraction from HTML: Application of a general learning approach. Proceedings of the Fifteenth Conference on Artificial Intelligence, (1998).

Google Scholar

[5] Bing Liu., Web Data Mining: Exploring Hyperlinks, Contents and Usage Data, Springer, (2009).

Google Scholar

[6] Baldi, P., Frasconi, P., & Smyth, P. Modeling the Internet and the Web. Probabilistic Methods and Algorithms. Chichester, UK: John Wiley & Sons. (2003).

Google Scholar

[7] Jiawei Han and Micheline Kamber. Data Mining: Concepts and Techniques, Morgan Kaufmann Publishers, (2006).

Google Scholar

[8] Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press, (2008).

DOI: 10.1007/s10791-009-9096-x

Google Scholar

[9] R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. ACM Press, New York, (1999).

Google Scholar

[10] Programming with POSIX threads, by D. Butenhof, Addison Wesley (1997).

Google Scholar