research-article

Watching inside the Screen: Digital Activity Monitoring for Task Recognition and Proactive Information Retrieval

Authors:

Giulio Jacucci,

Tuukka RuotsaloAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 109, Pages 1 - 23

https://doi.org/10.1145/3130974

Published: 11 September 2017 Publication History

Abstract

We investigate to what extent it is possible to infer a user’s work tasks by digital activity monitoring and use the task models for proactive information retrieval. Ten participants volunteered for the study, in which their computer screen was monitored and related logs were recorded for 14 days. Corresponding diary entries were collected to provide ground truth to the task detection method. We report two experiments using this data. The unsupervised task detection experiment was conducted to detect tasks using unsupervised topic modeling. The results show an average task detection accuracy of more than 70% by using rich screen monitoring data. The single-trial task detection and retrieval experiment utilized unseen user inputs in order to detect related work tasks and retrieve task-relevant information on-line. We report an average task detection accuracy of 95%, and the corresponding model-based document retrieval with Normalized Discounted Cumulative Gain of 98%. We discuss and provide insights regarding the types of digital tasks occurring in the data, the accuracy of task detection on different task types, and the role of using different data input such as application names, extracted keywords, and bag-of-words representations in the task detection process. We also discuss the implications of our results for ubiquitous user modeling and privacy.

References

[1]

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements That Don’T Add Up: Ad-hoc Retrieval Results Since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM ’09). ACM, New York, NY, USA, 601--610.

Digital Library

[2]

N.J. Belkin, R. N. Oddy, and H. M. Brooks. 1982. Ask for Information Retrieval: Part I.: Background and Theory. Journal of Documentation 38, 2 (1982), 61--71.

[3]

Daniel Billsus and Michael J. Pazzani. 2000. User Modeling for Adaptive News Access. User Modeling and User-Adapted Interaction 10, 2 (2000), 147--180.

Digital Library

[4]

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res. 3 (March 2003), 993--1022. http://dl.acm.org/citation.cfm?id=944919.944937

Digital Library

[5]

Pam Briggs, Elizabeth Churchill, Mark Levine, James Nicholson, Gary W. Pritchard, and Patrick Olivier. 2016. Everyday Surveillance. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16). ACM, New York, NY, USA, 3566--3573.

Digital Library

[6]

Katriina Byström and Preben Hansen. 2005. Conceptual Framework for Tasks in Information Studies. J. Am. Soc. Inf. Sci. Technol. 56, 10 (Aug. 2005), 1050--1061.

Digital Library

[7]

Sergey Chernov. 2008. Task Detection for Activity-based Desktop Search. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08). ACM, New York, NY, USA, 894--894.

Digital Library

[8]

Eun Kyoung Choe, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. 2014. Understanding Quantified-selfers’ Practices in Collecting and Exploring Personal Data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 1143--1152.

Digital Library

[9]

Elizabeth F. Churchill. 2014. Scrupulous, Scrutable, and Sumptuous: Personal Data Futures. interactions 21, 5 (Sept. 2014), 20--21.

Digital Library

[10]

Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science 41, 6 (1990), 391--407.

[11]

Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the Journey: A Query Log Analysis of Within-session Learning. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM ’14). ACM, New York, NY, USA, 223--232.

Digital Library

[12]

Gerhard Fischer. 2001. User Modeling in Human--Computer Interaction. User Modeling and User-Adapted Interaction 11, 1-2 (March 2001), 65--86.

Digital Library

[13]

Ramanathan Guha, Vineet Gupta, Vivek Raghunathan, and Ramakrishnan Srikant. 2015. User Modeling for a Personal Assistant. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM ’15). ACM, New York, NY, USA, 275--284.

Digital Library

[14]

Kirstie Hawkey and Kori M. Inkpen. 2006. Keeping Up Appearances: Understanding the Dimensions of Incidental Information Privacy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’06). ACM, New York, NY, USA, 821--830.

Digital Library

[15]

Eric Horvitz, Jack Breese, David Heckerman, David Hovel, and Koos Rommelse. 1998. The LumièRe Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI’98). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 256--265. http://dl.acm.org/citation.cfm?id=2074094.2074124

Digital Library

[16]

Wen Hua, Yangqiu Song, Haixun Wang, and Xiaofang Zhou. 2013. Identifying Users’ Topical Tasks in Web Search. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13). ACM, New York, NY, USA, 93--102.

Digital Library

[17]

Peter Ingwersen and Kalervo Järvelin. 2005. The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series). Springer-Verlag New York, Inc., Secaucus, NJ, USA.

Digital Library

[18]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-based Evaluation of IR Techniques. ACM Trans. Inf. Syst. 20, 4 (Oct. 2002), 422--446.

Digital Library

[19]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data As Implicit Feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’05). ACM, New York, NY, USA, 154--161.

Digital Library

[20]

Eunju Kim, Sumi Helal, and Diane Cook. 2010. Human Activity Recognition and Pattern Discovery. IEEE Pervasive Computing 9, 1 (Jan. 2010), 48--53.

Digital Library

[21]

Alfred Kobsa. 2007. Generic User Modeling Systems. Springer Berlin Heidelberg, Berlin, Heidelberg, 136--154.

Digital Library

[22]

Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2011. Identifying Task-based Sessions in Search Engine Query Logs. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM ’11). ACM, New York, NY, USA, 277--286.

Digital Library

[23]

Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2013. Discovering Tasks from Search Engine Query Logs. ACM Trans. Inf. Syst. 31, 3, Article 14 (Aug. 2013), 43 pages.

Digital Library

[24]

J. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics. University of California Press, Berkeley, Calif., 281--297. http://projecteuclid.org/euclid.bsmsp/1200512992

[25]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. CoRR abs/1301.3781 (2013). http://arxiv.org/abs/1301.3781

[26]

Feng Qiu and Junghoo Cho. 2006. Automatic Identification of User Interest for Personalized Search. In Proceedings of the 15th International Conference on World Wide Web (WWW ’06). ACM, New York, NY, USA, 727--736.

Digital Library

[27]

Andreas S. Rath, Didier Devaurs, and Stefanie N. Lindstaedt. 2009. UICO: An Ontology-based User Interaction Context Model for Automatic Task Detection on the Computer Desktop. In Proceedings of the 1st Workshop on Context, Information and Ontologies (CIAO ‘09). ACM, New York, NY, USA, Article 8, 10 pages.

Digital Library

[28]

Xuehua Shen, Bin Tan, and ChengXiang Zhai. 2005. Context-sensitive Information Retrieval Using Implicit Feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’05). ACM, New York, NY, USA, 43--50.

Digital Library

[29]

Fabrizio Silvestri. 2010. Mining Query Logs: Turning Search Usage Data into Knowledge. Found. Trends Inf. Retr. 4, 1--2 (Jan. 2010), 1--174.

Digital Library

[30]

Karen Sparck Jones. 1988. Document Retrieval Systems. Taylor Graham Publishing, London, UK, UK, Chapter A Statistical Interpretation of Term Specificity and Its Application in Retrieval, 132--142. http://dl.acm.org/citation.cfm?id=106765.106782

Digital Library

[31]

Bin Tan, Xuehua Shen, and ChengXiang Zhai. 2006. Mining Long-term Search History to Improve Search Accuracy. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’06). ACM, New York, NY, USA, 718--723.

Digital Library

[32]

Jaime Teevan, Susan T. Dumais, and Eric Horvitz. 2005. Personalizing Search via Automated Analysis of Interests and Activities. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’05). ACM, New York, NY, USA, 449--456.

Digital Library

[33]

Jaime Teevan, Susan T. Dumais, and Eric Horvitz. 2010. Potential for Personalization. ACM Trans. Comput.-Hum. Interact. 17, 1, Article 4 (April 2010), 31 pages.

Digital Library

[34]

Eran Toch, Yang Wang, and Lorrie Faith Cranor. 2012. Personalization and privacy: a survey of privacy risks and remedies in personalization-based systems. User Modeling and User-Adapted Interaction 22, 1 (2012), 203--220.

Digital Library

[35]

Manisha Verma and Emine Yilmaz. 2014. Entity Oriented Task Extraction from Query Logs. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM ’14). ACM, New York, NY, USA, 1975--1978.

Digital Library

[36]

Chirayu Wongchokprasitti, Jaakko Peltonen, Tuukka Ruotsalo, Payel Bandyopadhyay, Giulio Jacucci, and Peter Brusilovsky. 2015. User Model in a Box: Cross-System User Model Transfer for Resolving Cold Start Problems. Springer International Publishing, Cham, 289--301.

[37]

Dingqi Yang, Daqing Zhang, Longbiao Chen, and Bingqing Qu. 2015. NationTelescope: Monitoring and visualizing large-scale collective behavior in {LBSNs}. Journal of Network and Computer Applications 55 (2015), 170 -- 180.

[38]

Zack Zhu, Ulf Blanke, Alberto Calatroni, and Gerhard Tröster. 2013. Human Activity Recognition Using Social Media Data. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia (MUM ’13). ACM, New York, NY, USA, Article 21, 10 pages.

Digital Library

Cited By

Krieter P(2024)Recall for Good: Flexible Retrospective Mobile In-App Topic Tracking in a Privacy-Friendly Local-First ApproachProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703382(451-453)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703382
Routh TSaoda NNikseresht FBillah MGao JRajan VCampbell B(2024)ScreenSense: Screen Activity Detection in Real-World Environments with Indoor Light SensorsProceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3671127.3698167(22-32)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3671127.3698167
Vuong TJacucci GRuotsalo T(2024)Naturalistic Digital Behavior Predicts Cognitive AbilitiesACM Transactions on Computer-Human Interaction10.1145/366034131:3(1-32)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3660341
Show More Cited By

Index Terms

Watching inside the Screen: Digital Activity Monitoring for Task Recognition and Proactive Information Retrieval
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Proactive Information Retrieval via Screen Surveillance
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

We demonstrate proactive information retrieval via screen surveillance. A user's digital activities are continuously monitored by capturing all content on a user's screen using optical character recognition. This includes all applications and services ...
Bootstrapping activity modeling for ambient assisted living
ICSH'13: Proceedings of the 2013 international conference on Smart Health

In many societies, the age profile of the population is increasing, posing many challenges for societies, health services and carers. One response to this unfolding situation has been to direct research effort towards Ambient Assisted Living (AAL), ...
A multi-scale fuzzy entropy measure for anomaly detection in activities of daily living
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments

Anomaly detection in the Activities of Daily Living (ADL) of older adults is essential for healthcare management, to act to avoid prospective problems early and improve this group's quality of life. Once ADLs are recognised, the gathered information ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017

Accepted: 01 July 2017

Revised: 01 May 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Academy of Finland
Tekes

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
412
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)1

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Krieter P(2024)Recall for Good: Flexible Retrospective Mobile In-App Topic Tracking in a Privacy-Friendly Local-First ApproachProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703382(451-453)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703382
Routh TSaoda NNikseresht FBillah MGao JRajan VCampbell B(2024)ScreenSense: Screen Activity Detection in Real-World Environments with Indoor Light SensorsProceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3671127.3698167(22-32)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1145/3671127.3698167
Vuong TJacucci GRuotsalo T(2024)Naturalistic Digital Behavior Predicts Cognitive AbilitiesACM Transactions on Computer-Human Interaction10.1145/366034131:3(1-32)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3660341
R. Yousefi ZVuong TAlGhossein MRuotsalo TJaccuci GKaski S(2024)Entity Footprinting: Modeling Contextual User States via Digital Activity MonitoringACM Transactions on Interactive Intelligent Systems10.1145/364389314:2(1-27)Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3643893
Krieter PZerrer PPuschmann CGeise SBoratto LGena CMarras MGermanakos PPopescus E(2024)Following Topics Across All Apps and Media Formats: Mobile Keyword Tracking as a Privacy-Friendly Data Source in Mobile Media ResearchAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664879(126-131)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664879
Jadoon AYu CShi Y(2024)ContextMate: a context-aware smart agent for efficient data analysisCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-023-00144-76:3(199-227)Online publication date: 16-Apr-2024
https://doi.org/10.1007/s42486-023-00144-7
Bakhshizadeh MJilek CSchröder MMaus HDengel A(2024)Data Collection of Real-Life Knowledge Work in Context: The RLKWiC DatasetInformation Management10.1007/978-3-031-64359-0_22(277-290)Online publication date: 18-Jul-2024
https://doi.org/10.1007/978-3-031-64359-0_22
Harvey MBrazier D(2022)E-government information search by English-as-a Second Language speakers: The effects of language proficiency and document reading levelInformation Processing & Management10.1016/j.ipm.2022.10298559:4(102985)Online publication date: Jul-2022
https://doi.org/10.1016/j.ipm.2022.102985
Pellegrin FYücel ZMonden ALeelaprute P(2021)Task estimation for software company employees based on computer interaction logsEmpirical Software Engineering10.1007/s10664-021-10006-426:5Online publication date: 13-Jul-2021
https://dl.acm.org/doi/10.1007/s10664-021-10006-4
Krieter PPaternò FJacucci GRohs MSantoro C(2019)Can I record your screen?Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia10.1145/3365610.3365618(1-10)Online publication date: 26-Nov-2019
https://dl.acm.org/doi/10.1145/3365610.3365618
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents