Focused crawling using context graphs

M. Diligentit, F. M. Coetzee, S. Lawrence, C. L. Giles, M. Gori

Research output: Chapter in Book/Report/Conference proceedingConference contribution

363 Scopus citations

Abstract

Maintaining currency of search engine indices by exhaustive crawling is rapidly becoming impossible due to the increasing size and dynamic content of the web. Focused crawlers aim to search only the subset of the web related to a specific category, and offer a potential solution to the currency problem. The major problem in focused crawling is performing appropriate credit assignment to different documents along a crawl path, such that short-term gains are not pursued at the expense of less-obvious crawl paths that ultimately yield larger sets of valuable pages. To address this problem we present a focused crawling algorithm that builds a model for the context within which topically relevant pages occur on the web. This context model can capture typical link hierarchies within which valuable pages occur, as well as model content on documents that frequently cooccur with relevant pages. Our algorithm further leverages the existing capability of large search engines to provide partial reverse crawling capabilities. Our algorithm shows significant performance improvements in crawling efficiency over standard focused crawling.

Original languageEnglish (US)
Title of host publicationProceedings of the 26th International Conference on Very Large Data Bases, VLDB'00
Pages527-534
Number of pages8
StatePublished - Dec 1 2000
Event26th International Conference on Very Large Data Bases, VLDB 2000 - Cairo, Egypt
Duration: Sep 10 2000Sep 14 2000

Publication series

NameProceedings of the 26th International Conference on Very Large Data Bases, VLDB'00

Other

Other26th International Conference on Very Large Data Bases, VLDB 2000
CountryEgypt
CityCairo
Period9/10/009/14/00

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Information Systems
  • Software
  • Information Systems and Management

Cite this

Diligentit, M., Coetzee, F. M., Lawrence, S., Giles, C. L., & Gori, M. (2000). Focused crawling using context graphs. In Proceedings of the 26th International Conference on Very Large Data Bases, VLDB'00 (pp. 527-534). (Proceedings of the 26th International Conference on Very Large Data Bases, VLDB'00).