Graph based crawler seed selection

Shuyi Zheng; Pavel Dmitriev; C. Lee Giles

doi:10.1145/1526709.1526870

Graph based crawler seed selection

Shuyi Zheng, Pavel Dmitriev, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

15 Scopus citations

Abstract

This paper identifies and explores the problem of seed selection in a web-scale crawler. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a collection with more "good" and less "bad" pages. Based on the analysis of the graph structure of the web, we propose several seed selection algorithms. Effectiveness of these algorithms is proved by our experimental results on real web data. Copyright is held by the author/owner(s).

Original language	English (US)
Title of host publication	WWW'09 - Proceedings of the 18th International World Wide Web Conference
Pages	1089-1090
Number of pages	2
DOIs	https://doi.org/10.1145/1526709.1526870
State	Published - Dec 1 2009
Event	18th International World Wide Web Conference, WWW 2009 - Madrid, Spain Duration: Apr 20 2009 → Apr 24 2009

Other

Other	18th International World Wide Web Conference, WWW 2009
Country/Territory	Spain
City	Madrid
Period	4/20/09 → 4/24/09

All Science Journal Classification (ASJC) codes

Computer Networks and Communications

Access to Document

10.1145/1526709.1526870

Cite this

@inproceedings{69648615a0c242babe54e2c087f97eeb,

title = "Graph based crawler seed selection",

abstract = "This paper identifies and explores the problem of seed selection in a web-scale crawler. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a collection with more {"}good{"} and less {"}bad{"} pages. Based on the analysis of the graph structure of the web, we propose several seed selection algorithms. Effectiveness of these algorithms is proved by our experimental results on real web data. Copyright is held by the author/owner(s).",

author = "Shuyi Zheng and Pavel Dmitriev and Giles, {C. Lee}",

year = "2009",

month = dec,

day = "1",

doi = "10.1145/1526709.1526870",

language = "English (US)",

isbn = "9781605584874",

pages = "1089--1090",

booktitle = "WWW'09 - Proceedings of the 18th International World Wide Web Conference",

note = "18th International World Wide Web Conference, WWW 2009 ; Conference date: 20-04-2009 Through 24-04-2009",

}

TY - GEN

T1 - Graph based crawler seed selection

AU - Zheng, Shuyi

AU - Dmitriev, Pavel

AU - Giles, C. Lee

PY - 2009/12/1

Y1 - 2009/12/1

N2 - This paper identifies and explores the problem of seed selection in a web-scale crawler. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a collection with more "good" and less "bad" pages. Based on the analysis of the graph structure of the web, we propose several seed selection algorithms. Effectiveness of these algorithms is proved by our experimental results on real web data. Copyright is held by the author/owner(s).

AB - This paper identifies and explores the problem of seed selection in a web-scale crawler. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a collection with more "good" and less "bad" pages. Based on the analysis of the graph structure of the web, we propose several seed selection algorithms. Effectiveness of these algorithms is proved by our experimental results on real web data. Copyright is held by the author/owner(s).

UR - http://www.scopus.com/inward/record.url?scp=79960127814&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960127814&partnerID=8YFLogxK

U2 - 10.1145/1526709.1526870

DO - 10.1145/1526709.1526870

M3 - Conference contribution

AN - SCOPUS:79960127814

SN - 9781605584874

SP - 1089

EP - 1090

BT - WWW'09 - Proceedings of the 18th International World Wide Web Conference

T2 - 18th International World Wide Web Conference, WWW 2009

Y2 - 20 April 2009 through 24 April 2009

ER -

Graph based crawler seed selection

Abstract

Other

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this