Article

DP9: an OAI gateway service for web crawlers

Authors:

Xiaoming Liu,

Kurt Maly,

Mohammad Zubair,

Michael L. NelsonAuthors Info & Claims

JCDL '02: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries

Pages 283 - 284

https://doi.org/10.1145/544220.544284

Published: 14 July 2002 Publication History

Get Access

Abstract

Many libraries and databases are closed to general-purpose Web crawlers, and they expose their content only through their own search engines. At the same time many researchers attempt to locate technical papers through general-purpose Web search engines. DP9 is an open source gateway service that allows general search engines, (e.g. Google, Inktomi) to index OAI-compliant archives. DP9 does this by providing consistent URLs for repository records, and converting them to OAI queries against the appropriate repository when the URL is requested. This allows search engines that do not support the OAI protocol to index the "deep Web" contained within OAI compliant repositories.

References

[1]

M. K. Bergman. The Deep Web: Surfacing Hidden Value. Journal of Electronic Publishing, 7(1), 2001]]

Crossref

Google Scholar

[2]

M. Mahoui and S. J. Cunningham. Search Behavior in a Research-Oriented Digital Library. Proceedings of ECDL2001, Darmstadt, Germany, September 4--9, 2001, LNCS 2163, pp. 13--24]]

Digital Library

Google Scholar

[3]

C. Lagoze and H. Van de Sompel. The Open Archives Initiative: Building a low-barrier interoperability framework. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Roanoke VA, June 24-28, 2001, pp. 54--62]]

Digital Library

Google Scholar

[4]

X. Liu, K. Maly, M. Zubair, and M. L. Nelson. Arc - An OAI Service Provider for Digital Library Federation, D-Lib Magazine 7(4), April 2001]]

Google Scholar

[5]

M. Koster. The Web Robots Page. Available at http://info.webcrawler.com/mak/projects/robots/robots.html]]

Google Scholar

[6]

OAI Perl. Available at http://oai-perl.sourceforge.net/]]

Google Scholar

Cited By

View all

Kumar MBhatia RRattan D(2017)A survey of Web crawlers for information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.12187:6Online publication date: 7-Aug-2017
https://doi.org/10.1002/widm.1218
Kapidakis SHoussos NStamatis KKoutsourakis PMakedon FMariottini GKorn OMaglogiannis IMetsis V(2015)Flexible metadata mapping using OAI-PMHProceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments10.1145/2769493.2769531(1-2)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1145/2769493.2769531
Yi CLi C(2011)Localization Retrieval and Browse of a DSpace-Based Institutional Repository SystemAdvanced Materials Research10.4028/www.scientific.net/AMR.268-270.1401268-270(1401-1406)Online publication date: Jul-2011
https://doi.org/10.4028/www.scientific.net/AMR.268-270.1401
Show More Cited By

Index Terms

DP9: an OAI gateway service for web crawlers
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
2. Information systems
  1. Information systems applications
    1. Digital libraries and archives

Recommendations

Current challenges in web crawling
ICWE'13: Proceedings of the 13th international conference on Web Engineering

Web crawling, a process of collecting web pages in an automated manner, is the primary and ubiquitous operation used by a large number of web systems and agents starting from a simple program for website backup to a major web search engine. Due to an ...
Search Engine Coverage of the OAI-PMH Corpus

Having indexed much of the "surface" Web, search engines are now using various approaches to index the "deep"Web. At the same time, institutional repositories and digital libraries are adopting the Open Archives Initiative Protocol for Metadata ...
A framework for incremental deep web crawler based on URL classification
WISM'11: Proceedings of the 2011 international conference on Web information systems and mining - Volume Part II

With the Web grows rapidly, more and more data become available in the Deep Web.But users have to key in a set of keywords in order to access the pages from some web sites. Traditional search engines only index and retrieve Surface Web pages through ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

JCDL '02: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries

July 2002

448 pages

ISBN:1581135130

DOI:10.1145/544220

General Chair:
William Hersh
Oregon Health & Science University
,
Program Chair:
Gary Marchionini
University of North Carolina at Chapel Hill

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 July 2002

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

JCDL02

Sponsor:

JCDL02: Joint Conference on Digital Libraries 2002

July 14 - 18, 2002

Oregon, Portland, USA

Acceptance Rates

JCDL '02 Paper Acceptance Rate 69 of 240 submissions, 29%;

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Upcoming Conference

JCDL '24

Sponsor:
sigir
sigir

The 2024 ACM/IEEE Joint Conference on Digital Libraries

December 16 - 20, 2024

Hong Kong , China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
577
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kumar MBhatia RRattan D(2017)A survey of Web crawlers for information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.12187:6Online publication date: 7-Aug-2017
https://doi.org/10.1002/widm.1218
Kapidakis SHoussos NStamatis KKoutsourakis PMakedon FMariottini GKorn OMaglogiannis IMetsis V(2015)Flexible metadata mapping using OAI-PMHProceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments10.1145/2769493.2769531(1-2)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1145/2769493.2769531
Yi CLi C(2011)Localization Retrieval and Browse of a DSpace-Based Institutional Repository SystemAdvanced Materials Research10.4028/www.scientific.net/AMR.268-270.1401268-270(1401-1406)Online publication date: Jul-2011
https://doi.org/10.4028/www.scientific.net/AMR.268-270.1401
Kim HChoo CChen S(2010)Generating a meta-DL by federating search on OAI and non-OAI serversJournal of Intelligent Information Systems10.1007/s10844-009-0084-934:2(177-191)Online publication date: 1-Apr-2010
https://dl.acm.org/doi/10.1007/s10844-009-0084-9
Simeoni FYakici MNeely SCrestani F(2007)Metadata harvesting for content‐based distributed information retrievalJournal of the American Society for Information Science and Technology10.1002/asi.2069459:1(12-24)Online publication date: 3-Dec-2007
https://doi.org/10.1002/asi.20694
McCown FLiu XNelson MZubair M(2006)Search Engine Coverage of the OAI-PMH CorpusIEEE Internet Computing10.1109/MIC.2006.4110:2(66-73)Online publication date: 1-Mar-2006
https://dl.acm.org/doi/10.1109/MIC.2006.41
Masanés JMasanés J(2006)Archiving the Hidden WebWeb Archiving10.1007/978-3-540-46332-0_5(115-129)Online publication date: 2006
https://doi.org/10.1007/978-3-540-46332-0_5
Ntoulas AZerfos PCho JMarlino MSumner TShipman F(2005)Downloading textual hidden web content through keyword queriesProceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries10.1145/1065385.1065407(100-109)Online publication date: 7-Jun-2005
https://dl.acm.org/doi/10.1145/1065385.1065407
Maly KZubair MChilukamarri VKothari PBagherzadeh NValero MRamirez A(2005)GRID Based Federated Digital LibraryProceedings of the 2nd conference on Computing frontiers10.1145/1062261.1062281(97-105)Online publication date: 4-May-2005
https://dl.acm.org/doi/10.1145/1062261.1062281
Krowne AHalbert MChen HWactlar HChen CLim EChristel M(2004)Combined searching of web and oai digital library resourcesProceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries10.1145/996350.996428(343-344)Online publication date: 7-Jun-2004
https://dl.acm.org/doi/10.1145/996350.996428

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Current challenges in web crawling

Search Engine Coverage of the OAI-PMH Corpus

A framework for incremental deep web crawler based on URL classification