Article

How well does result relevance predict session satisfaction?

Authors:

Scott B. Huffman,

Michael HochsterAuthors Info & Claims

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 567 - 574

https://doi.org/10.1145/1277741.1277839

Published: 23 July 2007 Publication History

Abstract

Per-query relevance measures provide standardized, repeatable measurements of search result quality, but they ignore much of what users actually experience in a full search session. This paper examines how well we can approximate a user's ultimate session-level satisfaction using a simple relevance metric. We find that thisrelationship is surprisingly strong. By incorporating additional properties of the query itself, we construct a model which predicts user satisfaction even more accurately than relevance alone.

References

[1]

J. Allan, B. Carterette, and J. Lewis. When will information retrieval be \good enough"? In SIGIR'05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 433--440, New York, NY, USA, 2005. ACM Press.

Digital Library

[2]

D. Bilal. Children's use of the Yahooligans! Web search engine: Cognitive, physical, and affective behaviors on fact-based search tasks. J. Am. Soc. Inf. Sci., 51(7):646--665, 2000.

Digital Library

[3]

P. Borlund. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3), April 2003.

[4]

A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3--10, 2002.

Digital Library

[5]

C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40, New York, NY, USA, 2000. ACM Press.

Digital Library

[6]

C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25--32, New York, NY, USA, 2004. ACM Press.

Digital Library

[7]

C. W. Cleverdon. The signifcance of the Cranfeld tests on index languages. In SIGIR '91: Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, pages 3--12, New York, NY, USA, 1991. ACM Press.

Digital Library

[8]

B. J. Jansen and U. Pooch. A review of web searching studies and a framework for future research. J. Am. Soc. Inf. Sci. Technol., 52(3):235--246, 2001.

Digital Library

[9]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422--446, 2002.

Digital Library

[10]

D. Kahneman, P. P. Wakker, and R. Sarin. Back to Bentham? Explorations of experienced utility. The Quarterly Journal of Economics, 112(2):375--405, May 1997.

[11]

J. Reid. A task-oriented non-interactive evaluation methodology for information retrieval systems. Information Retrieval, 2(1):115--129, 2000.

Digital Library

[12]

D. E. Rose and D. Levinson. Understanding user goals in web search. In WWW '04: Proceedings of the 13th international conference on World Wide Web, pages 13--19, New York, NY, USA, 2004. ACM Press.

Digital Library

[13]

D. M. Russell and C. Grimes. Assigned and self-chosen tasks are not the same in web search. In HICSS '07: Proceedings of the 40th Annual International Conference on Systems and Software, 2007.

Digital Library

[14]

M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 162--169, New York, NY, USA, 2005. ACM Press.

Digital Library

[15]

A. Spink. A user-centered approach to evaluating human interaction with web search engines: an exploratory study. Inf. Process. Manage.,38(3):401--426, 2002.

Digital Library

[16]

J. Teevan, C. Alvarado, M. S. Ackerman, and D. R. Karger. The perfect search engine is not enough: A study of orienteering behavior in directed search. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 415--422, New York, NY, USA, 2004. ACM Press.

Digital Library

[17]

A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 11--18, New York, NY, USA, 2006. ACM Press.

Digital Library

[18]

A. H. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 225--231, New York, NY, USA, 2001. ACM Press.

Digital Library

[19]

E. M. Voorhees. Evaluation by highly relevant documents. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74--82, New York, NY, USA, 2001. ACM Press.

Digital Library

Cited By

Onodera NMaeda KOgawa THaseyama M(2024)Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender SystemsIEEE Access10.1109/ACCESS.2024.336069312(23626-23635)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3360693
Moffat AMackenzie J(2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
https://doi.org/10.1002/asi.24874
Mohian STang TTrinh TDang DCsallner CChandra SBlincoe KTonella P(2023)D2S2: Drag ’n’ Drop Mobile App Screen SearchProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613100(2177-2181)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613100
Show More Cited By

Index Terms

How well does result relevance predict session satisfaction?
1. Information systems
  1. Information retrieval

Recommendations

When does Relevance Mean Usefulness and User Satisfaction in Web Search?
SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

Relevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm ...
Investigating Cognitive Effects in Session-level Search User Satisfaction
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

User satisfaction is an important variable in Web search evaluation studies and has received more and more attention in recent years. Many studies regard user satisfaction as the ground truth for designing better evaluation metrics. However, most of the ...
Modeling user variance in time-biased gain
HCIR '12: Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval

Cranfield-style information retrieval evaluation considers variance in user information needs by evaluating retrieval systems over a set of search topics. For each search topic, traditional metrics model all users searching ranked lists in exactly the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

92
Total Citations
View Citations
2,526
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Onodera NMaeda KOgawa THaseyama M(2024)Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender SystemsIEEE Access10.1109/ACCESS.2024.336069312(23626-23635)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3360693
Moffat AMackenzie J(2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
https://doi.org/10.1002/asi.24874
Mohian STang TTrinh TDang DCsallner CChandra SBlincoe KTonella P(2023)D2S2: Drag ’n’ Drop Mobile App Screen SearchProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613100(2177-2181)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613100
Zobel J(2023)When Measurement MisleadsACM SIGIR Forum10.1145/3582524.358254056:1(1-20)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.1145/3582524.3582540
Hajian Hoseinabadi ACheshmehSohrabi M(2022)Proposing a New Combined Indicator for Measuring Search Engine Performance and Evaluating Google, Yahoo, DuckDuckGo, and Bing Search Engines based on Combined IndicatorJournal of Librarianship and Information Science10.1177/0961000622113857956:1(178-197)Online publication date: 8-Dec-2022
https://doi.org/10.1177/09610006221138579
Mohian SCsallner CMariani LCatolino GNagappan M(2022)PSDoodleProceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems10.1145/3524613.3527816(89-99)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524613.3527816
Mohian SCsallner CMariani LCatolino GNagappan M(2022)PSDoodleProceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems10.1145/3524613.3527807(84-88)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524613.3527807
Hassan SAmin AGordon ALee SHuenerfauth M(2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501986
Gao RGe YShah C(2022) FAIR : Fairness‐aware information retrieval evaluation Journal of the Association for Information Science and Technology10.1002/asi.2464873:10(1461-1473)Online publication date: 22-Mar-2022
https://doi.org/10.1002/asi.24648
YANG ZTAKAKU M(2021)The Query Satisfaction Prediction Considering Time Sequence時系列を考慮したクエリ満足度の推定Joho Chishiki Gakkaishi10.2964/jsik_2021_04731:2(343-354)Online publication date: 22-May-2021
https://doi.org/10.2964/jsik_2021_047
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents