research-article

Swipe and Tell: Using Implicit Feedback to Predict User Engagement on Tablets

Authors:

Klaas Nelissen,

Monique Snoeck,

Seppe Vanden Broucke,

Bart BaesensAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 36, Issue 4

Article No.: 35, Pages 1 - 36

https://doi.org/10.1145/3185153

Published: 13 June 2018 Publication History

Abstract

When content consumers explicitly judge content positively, we consider them to be engaged. Unfortunately, explicit user evaluations are difficult to collect, as they require user effort. Therefore, we propose to use device interactions as implicit feedback to detect engagement.

We assess the usefulness of swipe interactions on tablets for predicting engagement and make the comparison with using traditional features based on time spent.

We gathered two unique datasets of more than 250,000 swipes, 100,000 unique article visits, and over 35,000 explicitly judged news articles by modifying two commonly used tablet apps of two newspapers. We tracked all device interactions of 407 experiment participants during one month of habitual news reading.

We employed a behavioral metric as a proxy for engagement, because our analysis needed to be scalable to many users, and scanning behavior required us to allow users to indicate engagement quickly.

We point out the importance of taking into account content ordering, report the most predictive features, zoom in on briefly read content and on the most frequently read articles.

Our findings demonstrate that fine-grained tablet interactions are useful indicators of engagement for newsreaders on tablets. The best features successfully combine both time-based aspects and swipe interactions.

Supplementary Material

MP4 File (a35-nelissen.mp4)

Download
426.19 MB

References

[1]

E. Agichtein, E. Brill, and S. Dumais. 2006a. Improving web search ranking by incorporating user behavior information. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’06), 19--26.

Digital Library

[2]

E. Agichtein, E. Brill, S. Dumais, and R. Ragno. 2006b. Learning user interaction models for predicting web search result preferences. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’06), 3--10.

Digital Library

[3]

Ioannis Arapakis, M. Lalmas, B. Barla Cambazoglu, Mari-Carmen Marcos, and Joemon M. Jose. 2014b. User engagement in online news: Under the scope of sentiment, interest, affect, and gaze. J. Assoc. Inf. Sci. Technol. 65, 10 (2014), 1988--2005. 23301635

Digital Library

[4]

Ioannis Arapakis, M. Lalmas, and George Valkanas. 2014a. Understanding within-content engagement through pattern analysis of mouse gestures. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM’14). 1439--1448.

Digital Library

[5]

Bart Baesens, Veronique Van Vlasselaer, and Wouter Verbeke. 2015. Fraud Analytics Using Descriptive, Predictive, and Social Network Techniques: A Guide to Data Science for Fraud Detection. Wiley.

Digital Library

[6]

Nicholas J. Belkin, Michael Cole, and Jingjing Liu. 2009. A model for evaluation of interactive information retrieval. In Proceedings of the Special Interest Group on Information Retrieval Workshop on the Future of IR Evaluation (SIGIR’09), Vol. 43. 7--8.

[7]

Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16 (2002), 321--357.

[8]

M. Claypool, P. Le, M. Wased, and D. Brown. 2001. Implicit interest indicators. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’01). 33--40.

Digital Library

[9]

Michael Cole, Jingjing Liu, Nicholas J. Belkin, R. Bierig, Jacek Gwizdka, Chang Liu, J. Zhang, and Xiangmin Zhang. 2009. Usefulness as the criterion for evaluation of interactive information retrieval systems. In Proceedings of the Human Computer and Information Retrieval (HCIR’09). 1--4.

[10]

Elizabeth R. DeLong, David M. DeLong, and Daniel L. Clarke-Pearson. 1988. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 44, 3 (1988), 837--845.

[11]

A. Drutsa and P. Serdyukov. 2015. Future user engagement prediction and its application to improve the sensitivity of online experiments. In Proceedings of the International World Wide Web Conference (WWW’15). 256--266.

Digital Library

[12]

Georges Dupret and M. Lalmas. 2013. Absence time and user engagement: Evaluating ranking functions. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’13). 173--182.

Digital Library

[13]

Ashlee Edwards and Diane Kelly. 2016. Engagement in information search. In Why Engagement Matters: Cross-Disciplinary Perspectives of User Engagement in Digital Media, Heather L. O’Brien and Paul Cairns (Eds.). Springer International Publishing, 157--176.

[14]

Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst. 23, 2 (April 2005), 147--168.

Digital Library

[15]

Q. Guo and E. Agichtein. 2008. Exploring mouse movements for inferring query intent. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’08). 707--708.

Digital Library

[16]

Q. Guo and E. Agichtein. 2012. Beyond dwell time: Estimating document relevance from cursor movements and other post-click searcher behavior. In Proceedings of the International World Wide Web Conference (WWW’12). 569--578.

Digital Library

[17]

Q. Guo, H. Jin, D. Lagun, S. Yuan, and E. Agichtein. 2013b. Mining touch interaction data on mobile devices to predict web search result relevance. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’13). 153--162.

Digital Library

[18]

Q. Guo, H. Jin, D. Lagun, S. Yuan, and E. Agichtein. 2013a. Towards estimating web search result relevance from touch interactions on mobile devices. In CHI Extended Abstracts. 1821--1826.

Digital Library

[19]

Frank E. Harrell. 2015. Regression Modeling Strategies. Springer International Publishing.

[20]

Jeff Huang and Abdigani Diriye. 2012. Web user interaction mining from touch-enabled mobile devices. In Proceedings of the Human Computer and Information Retrieval (HCIR’12).

[21]

Jeff Huang, Ryen W. White, and Susan Dumais. 2011. No clicks, no problem: Using cursor movements to understand and improve search. In Proceedings of the Conference on Human Factors in Computing Systems (CHI’11). 1225--1234.

Digital Library

[22]

Jiepu Jiang, Daqing He, and James Allan. 2017a. Comparing in situ and multidimensional relevance judgments. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’17). 405--414.

Digital Library

[23]

Jiepu Jiang, Daqing He, Diane Kelly, and James Allan. 2017b. Understanding ephemeral state of relevance. In Proceedings of the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR’17). 137--146.

Digital Library

[24]

Diane Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retriev. 3, 1--2 (2009), 1--224.

Digital Library

[25]

J. A. Konstan, B. N. Miller, D. Maltz, J. L. Herlocker, L. R. Gordon, and J. Riedl. 1997. GroupLens: Applying collaborative filtering to usenet news. Commun. ACM 40, 3 (1997), 77--87.

Digital Library

[26]

D. Lagun, M. Ageev, Q. Guo, and E. Agichtein. 2014. Discovering common motifs in cursor movement data for improving web search. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’14). 183--192.

Digital Library

[27]

D. Lagun and M. Lalmas. 2016. Understanding and measuring user engagement and attention in online news reading. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’16). 113--122.

Digital Library

[28]

M. Lalmas, H. L. O’Brien, and E. Yom-Tov. 2014. Measuring user engagement. Synth. Lect. Inf. Concept. Retriev. Serv. 6, 4 (2014), 1--132.

Digital Library

[29]

H. J. Lee and Sung Joo Park. 2007. MONERS: A news recommender for the mobile web. Exp. Syst. Appl. 32, 1 (2007), 143--150.

[30]

Janette Lehmann, M. Lalmas, Elad Yom-tov, and Georges Dupret. 2012. Models of user engagement. In Proceedings of the User Modeling, Adaptation, and Personalization (UMAP’12). 164--175.

Digital Library

[31]

H. W. Lilliefors. 1967. On the kolmogorov-smirnov test for normality with mean and variance unknown. J. Am. Statist. Assoc. 62, 318 (1967), 399--402.

[32]

C. Liu, N. J. Belkin, and M. J. Cole. 2012. Personalization of search results using interaction behaviors in search sessions. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’12). 205--214.

Digital Library

[33]

Jingjing Liu and Nicholas J. Belkin. 2015. Personalizing information retrieval for multi-session tasks: Examining the roles of task stage, task type, and topic knowledge on the interpretation of dwell time as an indicator of document usefulness. J. Am. Soc. Inf. Sci. Technol. 66, 1 (May 2015), 58--81.

Digital Library

[34]

Jiahui Liu, Peter Dolan, and Er Pedersen. 2010. Personalized news recommendation based on click behavior. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’10). 31--40.

Digital Library

[35]

Yiqun Liu, Ye Chen, Jinhui Tang, Jiashen Sun, Min Zhang, Shaoping Ma, and Xuan Zhu. 2015. Different users, different opinions: Predicting search satisfaction with mouse movement information. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’15). 493--502.

Digital Library

[36]

Yiqun Liu, Xiaohui Xie, Chao Wang, Jian-Yun Nie, Min Zhang, and Shaoping Ma. 2016. Time-aware click model. ACM Trans. Inf. Syst. 35, 3, Article 16 (Dec. 2016), 24 pages.

Digital Library

[37]

Shiyang Lu, Tao Mei, Jingdong Wang, Jian Zhang, Zhiyong Wang, and Shipeng Li. 2014. Browse-to-search: Interactive exploratory search with visual entities. ACM Trans. Inf. Syst. 32, 4, Article 18 (Oct. 2014), 27 pages. 1046-8188

Digital Library

[38]

Naresh K. Malhotra, Sung S. Kim, and Ashutosh Patil. 2006. Common method in is research: A comparison of alternative approaches and a reanalysis of past research. Manage. Sci. 52, 12 (2006), 1865--1883.

Digital Library

[39]

Akhil Mathur, Nicholas D. Lane, and Fahim Kawsar. 2016. Engagement-aware computing: Modelling user engagement from mobile contexts. In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’16). 622--633.

Digital Library

[40]

Lori McCay-Peet, M. Lalmas, and Vidhya Navalpakkam. 2012. On saliency, affect and focused attention. In Proceedings of the Conference on Human Factors in Computing Systems (CHI’12). 541--551.

Digital Library

[41]

Masahiro Morita and Yoichi Shinoda. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’94). 272--281. http://dl.acm.org/citation.cfm?id=188490.188583

Digital Library

[42]

Vidhya Navalpakkam and Elizabeth Churchill. 2012. Mouse tracking: Measuring and predicting users’ experience of web-based content. In Proceedings of the Conference on Human Factors in Computing Systems (CHI’12). 2963--2972.

Digital Library

[43]

Heather L. O’Brien. 2011. Exploring user engagement in online news interactions. In Proceedings of the Association for Information Science and Technology Annual Meeting (ASIST’11), Vol. 48. 1--15.

[44]

H. L. O’Brien, R. Absar, and H. Halbert. 2013. Toward a model of mobile user engagement. In Proceedings of the Human Computer and Information Retrieval (HCIR’13).

[45]

H. L. O’Brien and M. Lebow. 2013. Mixed-methods approach to measuring user experience in online news interactions. J. Am. Soc. Inf. Sci. Technol. 64, 8 (2013), 1543--1556.

[46]

H. L. O’Brien and E. G. Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inf. Sci. Technol. 59, 6 (2008), 938--955.

[47]

H. L. O’Brien and E. G. Toms. 2010. The development and evaluation of a survey to measure user engagement. J. Am. Soc. Inf. Sci. Technol. 61, 1 (2010), 50--69.

[48]

Jeeyun Oh and S. Shyam Sundar. 2015. How does interactivity persuade? An experimental test of interactivity on cognitive absorption, elaboration, and attitudes. J. Commun. 65, 2 (2015), 213--236. 14602466

[49]

Philip M. Podsakoff, Scott B. MacKenzie, Jeong-Yeon Lee, and Nathan P. Podsakoff. 2003. Common method biases in behavioral research: A critical review of the literature and recommended remedies. J. Appl. Psychol. 88, 5 (2003), 879--903. arxiv:arXiv:1011.1669v3

[50]

Tefko Saracevic. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. J. Am. Soc. Inf. Sci. Technol. 58, 13 (2007), 1915--1933. arxiv:0803.1716

Digital Library

[51]

Bracha Shapira, Meirav Taieb-Maimon, and Anny Moskowitz. 2006. Study of the usefulness of known and new implicit indicators and their optimal combination for accurate inference of users interests. In Proceedings of the 2006 ACM Symposium on Applied Computing (SAC’06). 1118--1119.

Digital Library

[52]

Y. Song, Hao Ma, Hongning Wang, and Kuansan Wang. 2013a. Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance. In Proceedings of the International World Wide Web Conference (WWW’13). 1201--1212.

Digital Library

[53]

Y. Song, X. Shi, and X. Fu. 2013b. Evaluating and predicting user engagement change with degraded search relevance. In Proceedings of the International World Wide Web Conference (WWW’13). 1213--1223.

Digital Library

[54]

Maximilian Speicher and Martin Gaedke. 2013. TellMyRelevance&excl; Predicting the relevance of web search results from cursor interactions. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM’13). 1281--1290.

Digital Library

[55]

S. Shyam Sundar, Saraswathi Bellur, Jeeyun Oh, Haiyan Jia, and Hyang-Sook Kim. 2016. Theoretical importance of contingency in human-computer interaction: Effects of message interactivity on user engagement. Commun. Res. 43, 5 (2016), 595--625.

[56]

S. Shyam Sundar, Saraswathi Bellur, Jeeyun Oh, Qian Xu, and Haiyan Jia. 2014. User experience of on-screen interaction techniques: An experimental investigation of clicking, sliding, zooming, hovering, dragging, and flipping. Hum.-Comput. Interact. 29, 2 (2014), 109--152.

Digital Library

[57]

Roger Tourangeau, Lance J. Rips, and Kenneth Rasinski. 2000. The Psychology of Survey Response. Cambridge University Press.

[58]

Ryen W. White, Ian Ruthven, Joemon M. Jose, and C. J. Van Rijsbergen. 2005. Evaluating implicit feedback models using searcher simulations. ACM Trans. Inf. Syst. 23, 3 (July 2005), 325--361.

Digital Library

[59]

Yingying Wu, Yiqun Liu, Ning Su, Shaoping Ma, and Wenwu Ou. 2017. Predicting online shopping search satisfaction and user behaviors with electrodermal activity. In Proceedings of the International World Wide Web Conference (WWW’17). 855--856.

Digital Library

[60]

X. Yi, L. Hong, E. Zhong, N. Liu, and S. Rajan. 2014. Beyond clicks: dwell time for personalization. In Proceedings of the 8th ACM Conference on Recommender systems (RecSys’14). 113--120.

Digital Library

Cited By

Bhatele PBedekar M(2023)Survey on Smartphone Sensors and User Intent in Smartphone Usage2023 IEEE 8th International Conference for Convergence in Technology (I2CT)10.1109/I2CT57861.2023.10126192(1-9)Online publication date: 7-Apr-2023
https://doi.org/10.1109/I2CT57861.2023.10126192
Tian YZhou KPelleg D(2021)What and How long: Prediction of Mobile App EngagementACM Transactions on Information Systems10.1145/346430140:1(1-38)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3464301
Li CMarkov IRijke MZoghi M(2020)MergeDTSACM Transactions on Information Systems10.1145/341175338:4(1-28)Online publication date: 10-Sep-2020
https://dl.acm.org/doi/10.1145/3411753
Show More Cited By

Index Terms

Swipe and Tell: Using Implicit Feedback to Predict User Engagement on Tablets

Recommendations

B2B-Swipe: Swipe Gesture for Rectangular Smartwatches from a Bezel to a Bezel
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

We present B2B-Swipe, a single-finger swipe gesture for a rectangular smartwatch that starts at a bezel and ends at a bezel to enrich input vocabulary. There are 16 possible B2B-Swipes because a rectangular smartwatch has four bezels. Moreover, B2B-...
Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Zooming user interfaces are increasingly popular on mobile devices with touch screens. Swiping and pinching finger gestures anywhere on the screen manipulate the displayed portion of a page, and taps open objects within the page. This makes navigation ...
Use octopus launcher like your hands: joystick-based smartphone control solution for motor impaired people in electric wheelchairs
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

The rapid spread of smartphones based on touch interfaces is now playing a serious role in the digital divide to the physically disabled, especially to people with motor impairments. Motor impairments often result in muscle weakness, poor stamina, lack ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 36, Issue 4

October 2018

365 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/3211967

Editor:
Maarten de Rijke
University of Amsterdam, The Netherlands

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2018

Accepted: 01 February 2018

Revised: 01 December 2017

Received: 01 June 2017

Published in TOIS Volume 36, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

VLAIO

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
365
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)3

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhatele PBedekar M(2023)Survey on Smartphone Sensors and User Intent in Smartphone Usage2023 IEEE 8th International Conference for Convergence in Technology (I2CT)10.1109/I2CT57861.2023.10126192(1-9)Online publication date: 7-Apr-2023
https://doi.org/10.1109/I2CT57861.2023.10126192
Tian YZhou KPelleg D(2021)What and How long: Prediction of Mobile App EngagementACM Transactions on Information Systems10.1145/346430140:1(1-38)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3464301
Li CMarkov IRijke MZoghi M(2020)MergeDTSACM Transactions on Information Systems10.1145/341175338:4(1-28)Online publication date: 10-Sep-2020
https://dl.acm.org/doi/10.1145/3411753
Barbaro EGrua EMalavolta IStercevic MWeusthof Evan den Hoven J(2019)Modelling and predicting User Engagement in mobile applicationsData Science10.3233/DS-190027(1-17)Online publication date: 13-Nov-2019
https://doi.org/10.3233/DS-190027

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents