Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

An Efficient Parallel Crawler in Grid Environment

  • Conference paper
Grid and Cooperative Computing (GCC 2003)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 3032))

Included in the following conference series:

Abstract

As the size of the web grows, it is imperative to run multiple crawlers to gather data for search engines. In this paper we study the parallel crawling schema in grid environment. We propose and implement an advanced parallel crawler by introducing the techniques of dynamic partition, and evaluate the crawling schema based on parallel crawlers metrics. An experimental system built on Grid middleware has been tested in the real application.

Supported by Guangdong Key Laboratory of Computer Network under Grant 2002B60113.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed Web crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE 2002), San Jose, CA, February 26-March 1, pp. 357–368 (2002)

    Google Scholar 

  2. Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: A scalable fully distributed web crawler. In: Proc. Of 8th Australian World Wide Web Conference (2002)

    Google Scholar 

  3. Cho, J., Garcia-Molina, H.: Parallel crawlers. In: Proc. of the 11th International World– Wide Web Conference (2002)

    Google Scholar 

  4. Najork, M., Wiener, J.L.: Breadth-first search crawling yields high quality pages. In: Proc. of 10th International World Wide Web Conference, Hong Kong, China (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Network Research Center, South China University of Technology, 510640, Guangzhou, China

    Shoubin Dong, Xiaofeng Lu, Ling Zhang & Kejing He

Authors
  1. Shoubin Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaofeng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ling Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kejing He
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiatong University, 80 Dongcuan Road, 200240, Shanghai, China

    Minglu Li

  2. Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA

    Xian-He Sun

  3. Department. of Computer Science, Shanghai Jiaotong University, 1954 HuaShan Road, 200030, Shanghai, P.R. China

    Qian-ni Deng

  4. Department of Computer Science, College of Liberal Arts, University of Iowa, 52242, Iowa City, IA, USA

    Jun Ni

Rights and permissions

Reprints and permissions

Copyright information

© 2004 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Dong, S., Lu, X., Zhang, L., He, K. (2004). An Efficient Parallel Crawler in Grid Environment. In: Li, M., Sun, XH., Deng, Qn., Ni, J. (eds) Grid and Cooperative Computing. GCC 2003. Lecture Notes in Computer Science, vol 3032. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24679-4_48

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-24679-4_48

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-21988-0

  • Online ISBN: 978-3-540-24679-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation