research-article

Improving meeting summarization by focusing on user needs: a task-oriented evaluation

Authors:

Johanna D. MooreAuthors Info & Claims

IUI '09: Proceedings of the 14th international conference on Intelligent user interfaces

Pages 17 - 26

https://doi.org/10.1145/1502650.1502657

Published: 08 February 2009 Publication History

Abstract

Advances in multimedia technologies have enabled the creation of huge archives of audio-video recordings of meetings, and there is burgeoning interest in developing meeting browsers to help users better leverage these archives. A recent study has shown that extractive summaries provide a more efficient way of navigating meeting content than simply reading through the transcript and using the audio-video record, or navigating via keyword search (Murray, 2007). The extractive summary technique identifies informative dialogue acts to generate general purpose summaries. These summaries can still be lengthy. Recently, we have developed a decision-focused summarization system that presents only 1-2% of the recordings related to decision making. In this paper, we describe a task-based evaluation in which we compare the decision-focused summaries to the general purpose summaries. Our results indicate that the more focused summaries help users perform the decision debriefing task more effectively and improve perceived efficiency. In addition, this study also investigates the effect of automatic summaries and transcription on task effectiveness, report quality, and users' perceptions of task success.

References

[1]

S. Banerjee, C. Rose, and A. I. Rudnicky. The necessity of a meeting recording and playback system, and the benefit of topic-level annotations to meeting browsing. In Proceedings of the International Conference on Human-Computer Interaction, 2005.

Digital Library

[2]

R. Barzilay. Information Fusion for Multidocument Summarization: Paraphrasing and Generation. PhD thesis, Columbia University, 2003.

Digital Library

[3]

J. Carletta et al. The AMI meeting corpus: A pre-announcement. In S. Renals and S. Bengio, editors, Springer-Verlag Lecture Notes in Computer Science, volume 3869. Springer-Verlag, 2006.

Digital Library

[4]

A. H. Cremers, B. Hilhorst, and A. P. Vermeeren. What was discussed by whom, how, when and where? personalized browsing of annotated multimedia meeting recordings. In Proceedings of HCI, pages 1--10, 2005.

[5]

H. P. Edmundson. New methods in automatic extracting. Journal of the ACM, 16(2):264--285, 1968.

Digital Library

[6]

T. Hain, J. Dines, G. Garau, M. Karafiat, D. Moore, V. Wan, R. Ordelman, and S. Renals. Transcription of conference room meetings: An investigation. In Proceedings of Interspeech, 2005.

[7]

P. Hsueh and J. D. Moore. What decisions have you made?: Automatic decision detection in meeting conversations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 25--32, Rochester, New York, April 2007.

[8]

K. Koumpis, S. Renals, and M. Niranjan. Extractive summarization of voicemail using lexical and prosodic feature subset selection. In Proceedings of Eurospeech, pages 2377--2380, 2001.

[9]

J. Kupiec, J. Pedersen, and F. Chen. A trainable document summarizer. In Proceedings of the 18th annual international ACM SIGIR conference, pages 68--73, 1995.

Digital Library

[10]

C. Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out of ACL 2004, 2004.

[11]

A. Lisowska, A. Popescu-Belis, and S. Armstrong. User query analysis for the specification and evaluation of a dialogue processing and retrieval system. In Proceedings of LREC, pages 993--996, 2004.

[12]

I. Mani and E. Bloedorn. Machine learning of generic and user-focused summarization. In Proceedings of the Fifteenth National Conference on AI (AAAI), pages 821--826, 1998.

Digital Library

[13]

S. Maskey and J. Hirschberg. Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization. In Proceedings of Eurospeech, 2005.

[14]

M. Miekes, C. Müller, and M. Strube. Improving extractive dialogue summarization by utilizing human feedback. In Proceedings of AIAP, pages 627--632, 2007.

Digital Library

[15]

G. Murray. Automatic summarization of meeting. PhD thesis, University of Edinburgh, Scotland, UK, 2007.

[16]

G. Murray, S. Renals, and J. Carletta. Extractive summarization of meeting recordings. In Proceedings of Interspeech, 2005.

[17]

V. Pallotta, J. Niekrasz, and M. Purver. Collaborative and argumentative models of meeting discussions. In Workshop on Computational Models of Natural Arguments (CMNA) at the IJCAI, 2005.

[18]

V. Pallotta, V. Seretan, and M. Ailomaa. User requirements analysis for meeting information retrieval based on query elicitation. In Proceedings of ACL, 2007.

[19]

W. M. Post, A. H. Cremers, and O. B. Henkemans. A research environment for meeting behavior. In Proceedings of the 3rd Workshop on Social Intelligence Design, 2004.

[20]

R. Rienks, D. Heylen, and E. van der Weijden. Argument diagramming of meeting conversations. In Multimodal Multiparty Meeting Processing Workshop at the ICMI, 2005.

[21]

N. C. Romano and J. F. Nunamaker. Meeting analysis: Findings from research and practice. In Proceedings of HICSS-34. IEEE Computer Society, 2001.

Digital Library

[22]

S. Teufel and M. Moens. Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4):409--445, 2002.

Digital Library

[23]

P. Wellner, M. Flynn, and M. Guillemot. Browsing recorded meetings with ferret. In Proceedings of 1st conference on Machine Learning and Multi-Modal Interactions (MLMI'04). Springer-Verlag, 2004.

Digital Library

[24]

S. Whittaker, R. Laban, and S. Tucker. Analysing meeting records: An ethnographic study and technological implications. In Proceedings of MLMI, 2005.

Digital Library

[25]

K. Zechner. Automatic summarization of open-domain multiparty dialogues in diverse genres. Computational Linguistics, 28(4), 2002.

Digital Library

Cited By

Ito ANakano YNihei FSakato TIshii RFukayama ANakamura T(2023)Estimating and Visualizing Persuasiveness of Participants in Group DiscussionsJournal of Information Processing10.2197/ipsjjip.31.3431(34-44)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.34
Ranganathan ATamminaina SRaina G(2023)A Study Of Dialog Summarization Across Datasets And DomainsProceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval10.1145/3639233.3639245(196-202)Online publication date: 15-Dec-2023
https://dl.acm.org/doi/10.1145/3639233.3639245
Packowski SKulkarni RRichard SFaircloth GOnuţ IZulkernine F(2021)Using IBM watson services to process video to streamline business processes and improve customer experienceProceedings of the 31st Annual International Conference on Computer Science and Software Engineering10.5555/3507788.3507830(262-267)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.5555/3507788.3507830
Show More Cited By

Index Terms

Improving meeting summarization by focusing on user needs: a task-oriented evaluation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction paradigms
      1. Natural language interfaces

Recommendations

A complex network approach to text summarization

Automatic summarization of texts is now crucial for several information retrieval tasks owing to the huge amount of information available in digital media, which has increased the demand for simple, language-independent extractive summarization ...
Research on Automatic Summarization System Based on Topic Partition
WISM '09: Proceedings of the 2009 International Conference on Web Information Systems and Mining

Automatic summarization is an important issue in natural language processing and becomes more and more important in some domains such as web information retrieval. This paper describes an automatic summarization system based on topic partition: Firstly, ...
Social summarization method for feedback comments in online auction

Online auctions provide feedback comments for sellers written by buyers. Users select which seller to buy a product from by making reference to these comments. However, these feedback comments include a lot of courtesies in addition to being vast in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '09: Proceedings of the 14th international conference on Intelligent user interfaces

February 2009

522 pages

ISBN:9781605581682

DOI:10.1145/1502650

General Chairs:
Cristina Conati
University of British Columbia, Canada
,
Mathias Bauer
mineway GmbH, Germany
,
Program Chairs:
Nuria Oliver
Telefonica Research, Spain
,
Dan Weld
University of Washington, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 February 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI09

Sponsor:

IUI09: 14th International Conference on Intelligent User Interfaces

February 8 - 11, 2009

Florida, Sanibel Island, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
575
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ito ANakano YNihei FSakato TIshii RFukayama ANakamura T(2023)Estimating and Visualizing Persuasiveness of Participants in Group DiscussionsJournal of Information Processing10.2197/ipsjjip.31.3431(34-44)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.34
Ranganathan ATamminaina SRaina G(2023)A Study Of Dialog Summarization Across Datasets And DomainsProceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval10.1145/3639233.3639245(196-202)Online publication date: 15-Dec-2023
https://dl.acm.org/doi/10.1145/3639233.3639245
Packowski SKulkarni RRichard SFaircloth GOnuţ IZulkernine F(2021)Using IBM watson services to process video to streamline business processes and improve customer experienceProceedings of the 31st Annual International Conference on Computer Science and Software Engineering10.5555/3507788.3507830(262-267)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.5555/3507788.3507830
Nihei FNakano Y(2019)Exploring Methods for Predicting Important Utterances Contributing to Meeting SummarizationMultimodal Technologies and Interaction10.3390/mti30300503:3(50)Online publication date: 6-Jul-2019
https://doi.org/10.3390/mti3030050
Bahrainian SCrestani FShah CBelkin NByström KHuang JScholer F(2018)Augmentation of Human MemoryProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176399(150-159)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1145/3176349.3176399
McGregor MTang JLee CPoltrock SBarkhuus LBorges MKellogg W(2017)More to MeetingsProceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing10.1145/2998181.2998335(2208-2220)Online publication date: 25-Feb-2017
https://dl.acm.org/doi/10.1145/2998181.2998335
Nihei FNakano YTakase YNakano YAndré ENishida TMorency LBusso CPelachaud C(2016)Meeting extracts for discussion summarization based on multimodal nonverbal informationProceedings of the 18th ACM International Conference on Multimodal Interaction10.1145/2993148.2993160(185-192)Online publication date: 31-Oct-2016
https://dl.acm.org/doi/10.1145/2993148.2993160
Bothin AClough P(2014)A User Evaluation Study: Do Participants' Personal Notes Help Us to Summarise Meetings?Knowledge and Process Management10.1002/kpm.143921:2(122-133)Online publication date: 16-May-2014
https://doi.org/10.1002/kpm.1439
Rashid SCarenini GNg R(2013)A Visual Interface for Analyzing Text ConversationsEnabling Real-Time Business Intelligence10.1007/978-3-642-39872-8_7(93-108)Online publication date: 2013
https://doi.org/10.1007/978-3-642-39872-8_7
Bothin AClough P(2012)Participants' personal note-taking in meetings and its value for automatic meeting summarisationInformation Technology and Management10.1007/s10799-011-0112-713:1(39-57)Online publication date: 1-Mar-2012
https://dl.acm.org/doi/10.1007/s10799-011-0112-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents