Article

Industrial evaluation of a highly-accurate academic IR system

Authors:

Tahia Infantes-Morris,

Philip J. Bernhard,

Kevin L. Fox,

Gary J. Faulkner,

Kristina StriplingAuthors Info & Claims

CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

Pages 500 - 503

https://doi.org/10.1145/956863.956957

Published: 03 November 2003 Publication History

Get Access

Abstract

In this paper we report the results of an independent experimental evaluation of an information retrieval (IR) system developed at the Illinois Institute of Technology (IIT). The system, which is called the Advanced Information Retrieval Engine (AIRE), consists of a set of tools and utilities providing indexing, extraction, searching and visualization. We evaluated AIRE on three data sets from the Text REtrieval Conference (TREC) - TREC 8, 9 and 10. Overall, our results indicate that AIRE is a highly accurate IR system. Compared with results published by IIT, in our experiments AIRE consistently scored higher in recall. AIRE also scored higher in precision, but only for automatic tasks. In manual tasks, AIRE scored lower in precision in our experiments, but we attributed that to factors external to AIRE. Our final conclusion is that AIRE is a highly accurate IR system.

References

[1]

Ricardo Baeza-Yates, Berthier Ribeiro-Neto, and Berthier Ribiero-Neto, "Modern Information Retrieval," Pearson Education, May 1999.

Digital Library

Google Scholar

[2]

Gerald Kowalski, Information Retrieval Systems: Theory and Implementation, Kluwer Academic Publishers, Boston, 1997.

Digital Library

Google Scholar

[3]

Grossman, David A. and Frieder, Ophir, Information Retrieval: Algorithms and Heuristics, Kluwer Academic Publishers, 1998.

Digital Library

Google Scholar

[4]

Advaned Information Retrieval Engine (AIRE), http://www.ir.iit.edu/projects/AIRE.html.

Google Scholar

[5]

AIRE (Advanced Information Retrieval Engine) http://ir.iit.edu/abdur/research/aire/AIRE.html.

Google Scholar

[6]

McCabe, M. C., Chowdhury, A., Holmes, D. O., Grossman, D. A., Alford, K. L., and Frieder, O., "IIT at TREC-8: Improving Baseline Precision," NIST Special Publication 500--246: The Eighth Text Retrieval Conference (TReC 8).

Google Scholar

[7]

Chowdhury, A., Beitzel, S., Jensen, E., Sai-lee, M., Grossman, D. A., Frieder, O., McCabe, M. C., and Holmes, D. O., "IIT TREC-9 - Entity Based Feedback with Fusion," NIST Special Publication 500--249: The Ninth Text REtrieval Conference (TREC-9).

Google Scholar

[8]

Aljlayl, M., Beitzel, S., Jensen, E., Chowdhury, A., Holmes, D., Lee, M., Grossman, D., and Frieder, O., "IIT at TREC-10," NIST Special Publication 500--250: The Tenth Text REtrieval Conference (TREC 2001).

Google Scholar

[9]

Hawking, D., Voorhees, E., Craswell, N., and Bailey, P., "Overview of the TREC-8 Web Track," NIST Special Publication 500-246: The Eighth Text Retrieval Conference (TReC 8).

Google Scholar

[10]

Hawking, D., "Overview of the TREC-9 Web Track," NIST Special Publication 500--249: The Ninth Text REtrieval Conference (TREC-9).

Google Scholar

[11]

Voorhees, E. M., and Harman, D., "Overview of TREC 2001," NIST Special Publication 500--250: The Tenth Text REtrieval Conference (TREC 2001).

Google Scholar

[12]

Hawking, D., and Craswell, N., "Overview of the TREC-2001 Web Track," NIST Special Publication 500--250: The Tenth Text REtrieval Conference (TREC 2001).

Google Scholar

Cited By

View all

Cathey RJensen EBeitzel SFrieder OGrossman D(2007)Exploiting parallelism to support scalable hierarchical clusteringJournal of the American Society for Information Science and Technology10.1002/asi.2059658:8(1207-1221)Online publication date: 23-Apr-2007
https://doi.org/10.1002/asi.20596
Heard JWilberding JFrieder GFrieder OGrossman DKane L(2006)On mediated search of the united states holocaust memorial museum dataProceedings of the 6th international conference on Next Generation Information Technologies and Systems10.1007/11780991_4(38-46)Online publication date: 4-Jul-2006
https://dl.acm.org/doi/10.1007/11780991_4

Index Terms

Industrial evaluation of a highly-accurate academic IR system
1. Information systems
  1. Information retrieval

Recommendations

Query clustering and IR system detection: experiments on TREC data
RIAO '07: Large Scale Semantic Access to Content (Text, Image, Video, and Sound)

This paper investigates two aspects in this experiment. Linguistic techniques are used to categorize queries in a first step. This classification is then used to analyze systems performances in a TREC context. More precisely, we cluster TREC topics with ...
IR evaluation methods for retrieving highly relevant documents
SIGIR Test-of-Time Awardees 1978-2001

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable ...
Relevance dimensions in preference-based IR evaluation
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Evaluation of information retrieval (IR) systems has recently been exploring the use of preference judgments over two search result lists. Unlike the traditional method of collecting relevance labels per single result, this method allows to consider the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

November 2003

592 pages

ISBN:1581137230

DOI:10.1145/956863

General Chair:
Donald Kraft
Louisiana State University
,
Program Chairs:
Ophir Frieder
Illinois Institute of Technology
,
Joachim Hammer
University of Florida
,
Sajda Qureshi
University of Nebraska, Omaha
,
Len Seligman
The MITRE Corporation

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CIKM03

Sponsor:

CIKM03: 12th International Conference on Information and Knowledge Management

November 3 - 8, 2003

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
371
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Cathey RJensen EBeitzel SFrieder OGrossman D(2007)Exploiting parallelism to support scalable hierarchical clusteringJournal of the American Society for Information Science and Technology10.1002/asi.2059658:8(1207-1221)Online publication date: 23-Apr-2007
https://doi.org/10.1002/asi.20596
Heard JWilberding JFrieder GFrieder OGrossman DKane L(2006)On mediated search of the united states holocaust memorial museum dataProceedings of the 6th international conference on Next Generation Information Technologies and Systems10.1007/11780991_4(38-46)Online publication date: 4-Jul-2006
https://dl.acm.org/doi/10.1007/11780991_4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Query clustering and IR system detection: experiments on TREC data

IR evaluation methods for retrieving highly relevant documents

Relevance dimensions in preference-based IR evaluation