column

Accurately Interpreting Clickthrough Data as Implicit Feedback

Authors:

Thorsten Joachims,

Helene Hembrooke,

Geri GayAuthors Info & Claims

ACM SIGIR Forum, Volume 51, Issue 1

Pages 4 - 11

https://doi.org/10.1145/3130332.3130334

Published: 02 August 2017 Publication History

Abstract

This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average.

References

[1]

R. Almeida and V. Almeida. A community-aware search engine. In Proceedings of the World Wide Web Conference (WWW), 2004.

Digital Library

[2]

B. Bartell, G. Cottrell, and R. Belew. Automatic combination of multiple ranked retrieval systems. In Annual ACM SIGIR Conf. on Research and Development in Information Retrieval (SIGIR), pages 173--181, 1994.

[3]

R. Belew. Finding Out About. Cambridge, 2000.

[4]

J. Boyan, D. Freitag, and T. Joachims. A machine learning architecture for optimizing web search engines. In AAAI Workshop on Internet Based Information Systems, pages 1--8, August 1996.

[5]

A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3--10, 2002.

Digital Library

[6]

M. Claypool, P. Le, M. Waseda, and D. Brown. Implicit interest indicators. In International Conference on Intelligent User Interfaces (IUI), pages 33--40, 2001.

Digital Library

[7]

W. Cohen, R. Shapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243--270, 1999.

[8]

S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve the search experiences. In Talk presented at SIGIR03 Workshop on Implicit Measures of User Interests and Preferences, 2003.

[9]

N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Transactions on Information Systems, 7(3):183--204, 1989.

Digital Library

[10]

J. Goldberg, M. Stimson, M. Lewenstein, M. Scott, and A. Wichansky. Eye-tracking in web search tasks: design implications. In Proceedings of the Eye tracking Research and Applications Symposium (ETRA), pages 51--58, 2002.

Digital Library

[11]

L. Granka, T. Joachims, and G. Gay. Eye-tracking analysis of user behavior in www search. In ACM Conference on Research and Development in Information Retrieval (SIGIR), 2004.

Digital Library

[12]

T. Halverson and A. Hornof. Link colors guide a search. In ACM Conference on Computer-Human Interaction (CHI), 2004.

Digital Library

[13]

S. Holland, M. Ester, and W. Kieling. Preference mining: A novel approach on mining user preferences for personalized applications. In Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), pages 204--216, 2003.

[14]

T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2002.

Digital Library

[15]

M. Just and P. Carpenter. A theory of reading: From eye fixations to comprehension. Psychological Review, 87:329--354, 1980.

[16]

D. Kelly and N. Belkin. Display time as implicit feedback: Understanding task effects. In ACM International Conference on Research and Development in Information Retrieval (SIGIR), pages 377--384, 2004.

Digital Library

[17]

D. Kemp and K. Ramamohanarao. Long-term learning for web search engines. In European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), pages 263--274, 2002.

[18]

K. Kloeckner, N. Wirschum, and A. Jameson. Depth- and breadth-first processing of search result lists. In ACM Conference on Computer-Human Interaction, 2004.

Digital Library

[19]

C. Lankford. Gazetracker: software designed to facilitate eye movement analysis. In Proceedings of Eye Tracking Research & Applications, pages 51--55, 2000.

Digital Library

[20]

M. Morita and Y. Shinoda. Information filtering based on user behavior analysis and best match text retrieval. In ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 272--281, 1994.

[21]

B. Pan, H. Hembrooke, G. Gay, L. Granka, M. Feusner, and J. Newman. The determinants of web page viewing behavior: An eye tracking study. In S. Spencer, editor, Proceedings of Eye Tracking Research & Applications. ACM, New York, 2004.

Digital Library

[22]

F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. In ACM SIGKDD International Conference On Knowledge Discovery and Data Mining (KDD), 2005.

Digital Library

[23]

K. Rayner. Eye movements in reading and information processing. Psychological Bulletin, 124:372--252, 1998.

[24]

E. Russo and F. LeClerc. An eye-fixation analysis of choice processes for consumer nondurables. Journal of Consumer Research, 21(2):274--290, 1994.

[25]

J. Salogarvi, I. Kojo, S. Jaana, and S. Kaski. Can relevance be inferred from eye movements in information retrieval? In Proceedings of the Workshop on Self-Organizing Maps, pages 261--266, 2003.

Cited By

León-Garrido ABarroso-Osuna JLLorente-Cejudo C(2025)Consistencia interevaluador en la evaluación de una app móvil para Educación Musical Básica: estudio pilotoInnovación educativa10.15304/ie.35.10389Online publication date: 14-Feb-2025
https://doi.org/10.15304/ie.35.10389
Alvarez JMastropietro ARuggieri SNejdl WAuer SKarras OCha MMoens MNajork M(2025)The Initial Screening Order ProblemProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703497(165-174)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703497
Kim MJoo MHan K(2025)Understanding the Differences in an AI-Based Creativity Support Tool Between Creativity Types in Fashion DesignInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2448484(1-14)Online publication date: 14-Jan-2025
https://doi.org/10.1080/10447318.2024.2448484
Show More Cited By

Index Terms

Accurately Interpreting Clickthrough Data as Implicit Feedback
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Accurately interpreting clickthrough data as implicit feedback
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks ...
Impact of query intent and search context on clickthrough behavior in sponsored search

Implicit feedback techniques may be used for query intent detection, taking advantage of user behavior to understand their interests and preferences. In sponsored search, a primary concern is the user's interest in purchasing or utilizing a commercial ...
Click data as implicit relevance feedback in web search

Search sessions consist of a person presenting a query to a search engine, followed by that person examining the search results, selecting some of those search results for further review, possibly following some series of hyperlinks, and perhaps ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum

ACM SIGIR Forum Volume 51, Issue 1

June 2017

73 pages

ISSN:0163-5840

DOI:10.1145/3130332

Editors:
Claudia Hauff
Delft University of Technology. The Netherlands
,
Craig Macdonald
University of Glasgow, Glasgow, UK

Issue’s Table of Contents

Copyright © 2017 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 August 2017

Published in SIGIR Volume 51, Issue 1

Check for updates

Author Tags

Qualifiers

Column

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

150
Total Citations
View Citations
1,162
Total Downloads

Downloads (Last 12 months)110
Downloads (Last 6 weeks)13

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

León-Garrido ABarroso-Osuna JLLorente-Cejudo C(2025)Consistencia interevaluador en la evaluación de una app móvil para Educación Musical Básica: estudio pilotoInnovación educativa10.15304/ie.35.10389Online publication date: 14-Feb-2025
https://doi.org/10.15304/ie.35.10389
Alvarez JMastropietro ARuggieri SNejdl WAuer SKarras OCha MMoens MNajork M(2025)The Initial Screening Order ProblemProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703497(165-174)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703497
Kim MJoo MHan K(2025)Understanding the Differences in an AI-Based Creativity Support Tool Between Creativity Types in Fashion DesignInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2448484(1-14)Online publication date: 14-Jan-2025
https://doi.org/10.1080/10447318.2024.2448484
Cao SZheng QZhan ZYang YLv HZheng DZhang W(2025)FlyCache: Recommendation-Driven Edge Caching Architecture for Full Life Cycle of Video StreamingDigital Communications and Networks10.1016/j.dcan.2025.01.001Online publication date: Jan-2025
https://doi.org/10.1016/j.dcan.2025.01.001
Shen YWang SShen YTan SDong YQin WZhuang Y(2024)Evaluating the Usability of mHealth Apps: An Evaluation Model Based on Task Analysis Methods and Eye Movement DataHealthcare10.3390/healthcare1213131012:13(1310)Online publication date: 30-Jun-2024
https://doi.org/10.3390/healthcare12131310
Pradel F(2024)Influence of hate speech about refugees in search algorithms on political attitudes: An online experimentNew Media & Society10.1177/14614448241244735Online publication date: 29-Apr-2024
https://doi.org/10.1177/14614448241244735
Locke HThomas AStotz NMetz A(2024)Information about Tummy Time in infants available on the InternetHealth Education Journal10.1177/0017896924124104383:3(293-308)Online publication date: 30-Mar-2024
https://doi.org/10.1177/00178969241241043
Wan SWu YLiu YXiao LGuo M(2024)A Recommendation Approach Based on Heterogeneous Network and Dynamic Knowledge GraphInternational Journal of Intelligent Systems10.1155/2024/41694022024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/4169402
Tavakoli LTrippas JZamani HScholer FSanderson M(2024)Online and Offline Evaluation in Search ClarificationACM Transactions on Information Systems10.1145/368178643:1(1-30)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3681786
Lin SZhou SChen JFeng YShi QChen CLi YWang C(2024)ReCRec: Reasoning the Causes of Implicit Feedback for Debiased RecommendationACM Transactions on Information Systems10.1145/367227542:6(1-26)Online publication date: 18-Oct-2024
https://dl.acm.org/doi/10.1145/3672275
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents