Article

Summarization evaluation using relative utility

Authors:

Dragomir R. Radev,

Daniel TamAuthors Info & Claims

CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

Pages 508 - 511

https://doi.org/10.1145/956863.956960

Published: 03 November 2003 Publication History

Get Access

Abstract

We present a series of experiments to demonstrate the validity of Relative Utility (RU) as a measure for evaluating extractive summarizers. RU is applicable in both single-document and multi-document summarization, is extendable to arbitrary compression rates with no extra annotation effort, and takes into account both random system performance and interjudge agreement. Our results using the JHU summary corpus indicate that RU is a reasonable and often superior alternative to several common evaluation metrics.

References

[1]

Hongyan Jing, Kathleen McKeown, Regina Barzilay, and Michael Elhadad. Summarization Evaluation Methods: Experiments and Analysis. In Intelligent Text Summarization. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-06, pages 60--68, Standford (CA), USA, March 23-25 1998. The AAAI Press.

Google Scholar

[2]

Jade Goldstein, Mark Kantrowitz, Vibhu O. Mittal, and Jaime G. Carbonell. Summarizing text documents: Sentence selection and evaluation metrics. In Research and Development in Information Retrieval, pages 121--128, Berkeley, California, 1999.

Digital Library

Google Scholar

[3]

Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In ANLP/NAACL Workshop on Summarization, Seattle, WA, April 2000.

Digital Library

Google Scholar

[4]

RobertL. Donaway, KevinW. Drummey, and LauraA. Mather. A Comparison of Rankings Produced by Summarization Evaluation Measures. In Proceedings of the Workshop on Automatic Summarization, ANLP-NAACL2000, pages 69--78. Association for Computational Linguistics, 30 April 2000.

Digital Library

Google Scholar

Cited By

View all

M RPankaj D(2023)Challenges in Creating Text Summarization Models in Malayalam: A Study2023 International Conference on Control, Communication and Computing (ICCC)10.1109/ICCC57789.2023.10165363(1-6)Online publication date: 19-May-2023
https://doi.org/10.1109/ICCC57789.2023.10165363
Mridha MLima ANur KDas SHasan MKabir M(2021)A Survey of Automatic Text Summarization: Progress, Process and ChallengesIEEE Access10.1109/ACCESS.2021.31297869(156043-156070)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3129786
Lloret EPlaza LAker A(2018)The challenging task of summary evaluationLanguage Resources and Evaluation10.1007/s10579-017-9399-252:1(101-148)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s10579-017-9399-2
Show More Cited By

Index Terms

Summarization evaluation using relative utility
1. Information systems
  1. Information retrieval

Recommendations

Sentiment diversification for short review summarization
WI '17: Proceedings of the International Conference on Web Intelligence

With the abundance of reviews published on the Web about a given product, consumers are looking for ways to view major opinions that can be presented in a quick and succinct way. Reviews contain many different opinions, making the ability to show a ...
Sentence Retrieval with Sentiment-specific Topical Anchoring for Review Summarization
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management

We propose Topic Anchoring-based Review Summarization (TARS), a two-step extractive summarization method, which creates review summaries from the sentences that represent the most important aspects of a review. In the first step, the proposed method ...
Evaluation of Cross Domain Text Summarization
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Extractive-abstractive hybrid summarization can generate readable, concise summaries for long documents. Extraction-then-abstraction and extraction-with-abstraction are two representative approaches to hybrid summarization. But their general performance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

November 2003

592 pages

ISBN:1581137230

DOI:10.1145/956863

General Chair:
Donald Kraft
Louisiana State University
,
Program Chairs:
Ophir Frieder
Illinois Institute of Technology
,
Joachim Hammer
University of Florida
,
Sajda Qureshi
University of Nebraska, Omaha
,
Len Seligman
The MITRE Corporation

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CIKM03

Sponsor:

CIKM03: 12th International Conference on Information and Knowledge Management

November 3 - 8, 2003

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
528
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

M RPankaj D(2023)Challenges in Creating Text Summarization Models in Malayalam: A Study2023 International Conference on Control, Communication and Computing (ICCC)10.1109/ICCC57789.2023.10165363(1-6)Online publication date: 19-May-2023
https://doi.org/10.1109/ICCC57789.2023.10165363
Mridha MLima ANur KDas SHasan MKabir M(2021)A Survey of Automatic Text Summarization: Progress, Process and ChallengesIEEE Access10.1109/ACCESS.2021.31297869(156043-156070)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3129786
Lloret EPlaza LAker A(2018)The challenging task of summary evaluationLanguage Resources and Evaluation10.1007/s10579-017-9399-252:1(101-148)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s10579-017-9399-2
Yang SLu WYang DLi XWu CWei B(2017)KeyphraseDSNeurocomputing10.1016/j.neucom.2016.10.052224:C(58-70)Online publication date: 8-Feb-2017
https://dl.acm.org/doi/10.1016/j.neucom.2016.10.052
Gambhir MGupta V(2017)Recent automatic text summarization techniquesArtificial Intelligence Review10.1007/s10462-016-9475-947:1(1-66)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1007/s10462-016-9475-9
Polepalli Ramesh BSethi RYu H(2015)Figure-Associated Text Summarization and EvaluationPLOS ONE10.1371/journal.pone.011567110:2(e0115671)Online publication date: 2-Feb-2015
https://doi.org/10.1371/journal.pone.0115671
Sarkar K(2014)Multilingual Summarization ApproachesComputational Linguistics10.4018/978-1-4666-6042-7.ch009(158-177)Online publication date: 2014
https://doi.org/10.4018/978-1-4666-6042-7.ch009
Sarkar K(2014)Multilingual Summarization ApproachesInnovative Document Summarization Techniques10.4018/978-1-4666-5019-0.ch011(257-276)Online publication date: 2014
https://doi.org/10.4018/978-1-4666-5019-0.ch011
Ferreira RLins RFreitas FSimske SRiss MSimske SRönnau S(2014)A new sentence similarity assessment measure based on a three-layer sentence representationProceedings of the 2014 ACM symposium on Document engineering10.1145/2644866.2644881(25-34)Online publication date: 16-Sep-2014
https://dl.acm.org/doi/10.1145/2644866.2644881
Palshikar GDeshpande SAthiappan G(2012)Combining summaries using unsupervised rank aggregationProceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II10.1007/978-3-642-28601-8_32(378-389)Online publication date: 11-Mar-2012
https://dl.acm.org/doi/10.1007/978-3-642-28601-8_32
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Sentiment diversification for short review summarization

Sentence Retrieval with Sentiment-specific Topical Anchoring for Review Summarization

Evaluation of Cross Domain Text Summarization