Article

Evaluating web search result summaries

Authors:

Shao Fen Liang,

Siobhan Devlin,

John TaitAuthors Info & Claims

ECIR'06: Proceedings of the 28th European conference on Advances in Information Retrieval

Pages 96 - 106

https://doi.org/10.1007/11735106_10

Published: 10 April 2006 Publication History

Abstract

The aim of our research is to produce and assess short summaries to aid users' relevance judgements, for example for a search engine result page. In this paper we present our new metric for measuring summary quality based on representativeness and judgeability, and compare the summary quality of our system to that of Google. We discuss the basis for constructing our evaluation methodology in contrast to previous relevant open evaluations, arguing that the elements which make up an evaluation methodology: the tasks, data and metrics, are interdependent and the way in which they are combined is critical to the effectiveness of the methodology. The paper discusses the relationship between these three factors as implemented in our own work, as well as in SUMMAC/MUC/DUC.

References

[1]

Afantenos, S., Karkaletsis, V. and Stamatopoulos, P.: Summarization from medical documents: a survey. Artificial Intelligence in Medicine, 33, 2, February (2005), 157-177.

Digital Library

Google Scholar

[2]

Berger, A. and Mittal, V.O.: Query-Relevant summarisation using FAQs. ACL (2000) 294-301.

Digital Library

Google Scholar

[3]

Borko, H. and Bernier, C.L.: Abstracting concepts and methods. Academic Press, San Diego, Ca., USA (1975)

Google Scholar

[4]

Chinchor, N. Hirschman, L. and Lewis, D.D.: Evaluating message understanding systems: An analysis of the third message understanding conference. Association for Computation Linguistics, 19, 3, (1993) 409-449.

Digital Library

Google Scholar

[5]

Chinchor, N.: MUC-3 Evaluation metrics. Proceedings of third message understanding conference (1991) 17-24.

Digital Library

Google Scholar

[6]

Harman, D. and Over P.: The effects of human variation in DUC summarisation evaluation. Proceedings of the ACL-04 Workshop in Text Summarization Branches Out (Barcelona, Spain, July, (2004) 10-17.

Google Scholar

[7]

Liang, S.F., Devlin, S. and Tait, J.: Poster: Using query term order for result summarisation. SIGIR'05, Brazil (2005) 629-630.

Digital Library

Google Scholar

[8]

Lin, C.Y.: ROUGE: a Package for Automatic Evaluation of Summaries. Proceedings of the Workshop on Text Summarization Branches Out, Barcelona, Spain, (2004) 25-26, July.

Google Scholar

[9]

Mani, I., Firmin, T. and Sundheim, B.: The TIPSTER SUMMAC text summarisation evaluation. Proceedings of the ninth conference on European chapter of the Association ofr Computational Linguistics, Bergen, Norway (1999) 77-85.

Digital Library

Google Scholar

[10]

Mani, I.: Automatic Summarization. John Benjamins, Amsterdam (2001).

Google Scholar

[11]

Pagano, R.R.: Understanding statistics in the behavioural sciences. Wadsworth/Thomson Learning, USA (2001).

Google Scholar

[12]

Sparck Jones, C. and Galliers, J.R.: Evaluating natural language processing systems: an analysis and review. Springer, New York (1996).

Digital Library

Google Scholar

[13]

Tipster text phase III 18-month workshop notes, (1998), May, (1998) Fairfax, VA.

Google Scholar

[14]

Voorhees, E.M.: Variations in Relevance Judgements and the Measurement of Retrieval Effectiveness. Information Processing & Management, 36, 5, (2000) 697-716, September.

Digital Library

Google Scholar

Cited By

View all

Alshomary MDüsterhus NWachsmuth HHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Extractive Snippet Generation for ArgumentsProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401186(1969-1972)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401186
Ageev MLagun DAgichtein EJones GSheridan PKelly Dde Rijke MSakai T(2013)Improving search result summaries by using searcher behavior dataProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484093(13-22)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484093
Kanungo TOrr D(2009)Predicting the readability of short web summariesProceedings of the Second ACM International Conference on Web Search and Data Mining10.1145/1498759.1498827(202-211)Online publication date: 9-Feb-2009
https://dl.acm.org/doi/10.1145/1498759.1498827
Show More Cited By

Recommendations

Evaluating leading web search engines on children's queries
HCII'11: Proceedings of the 14th international conference on Human-computer interaction: users and applications - Volume Part IV

This study compared retrieved results, relevance ranking, and overlap across Google, Yahoo!, Bing, Yahoo Kids!, and Ask Kids on 15 queries constructed by middle school children. Queries included one word, two words, and multiple words/phrases/natural ...
Evaluating and Analyzing Click Simulation in Web Search
ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

We evaluate and analyze the quality of click models with respect to their ability to simulate users' click behavior. To this end, we propose distribution-based metrics for measuring the quality of click simulation in addition to metrics that directly ...
Web search result summarization: title selection algorithms and user satisfaction
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Eye tracking experiments have shown that titles of Web search results play a crucial role in guiding a user's search process. We present a machine-learned algorithm that trains a boosted tree to pick the most relevant title for a Web search result. We ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ECIR'06: Proceedings of the 28th European conference on Advances in Information Retrieval

April 2006

582 pages

ISBN:3540333479

Editors:
Mounia Lalmas
Queen Mary, University of London, London, UK
,
Andy MacFarlane
Department of Information Science, City University, Northampton Square, London, UK
,
Stefan Rüger
Knowledge Media Institute, The Open University, Northampton Square, Milton Keynes, UK
,
Anastasios Tombros
Knowledge Media Institute, Queen Mary University of London, Northampton Square, Milton Keynes, UK
,
Theodora Tsikrika
Knowledge Media Institute, CWI, Northampton Square, Amsterdam, The Netherlands

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 10 April 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alshomary MDüsterhus NWachsmuth HHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Extractive Snippet Generation for ArgumentsProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401186(1969-1972)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401186
Ageev MLagun DAgichtein EJones GSheridan PKelly Dde Rijke MSakai T(2013)Improving search result summaries by using searcher behavior dataProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484093(13-22)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484093
Kanungo TOrr D(2009)Predicting the readability of short web summariesProceedings of the Second ACM International Conference on Web Search and Data Mining10.1145/1498759.1498827(202-211)Online publication date: 9-Feb-2009
https://dl.acm.org/doi/10.1145/1498759.1498827
Lalmas MRüger STsikrika TYavlinsky A(2006)Progress in information retrievalProceedings of the 28th European conference on Advances in Information Retrieval10.1007/11735106_1(1-11)Online publication date: 10-Apr-2006
https://dl.acm.org/doi/10.1007/11735106_1

Abstract

References

Cited By

Recommendations

Evaluating leading web search engines on children's queries

Evaluating and Analyzing Click Simulation in Web Search

Web search result summarization: title selection algorithms and user satisfaction

Comments

Information

Published In

Sponsors

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Share

Share this Publication link

Share on social media

Affiliations