research-article

Public Access

Reading Protocol: Understanding what has been Read in Interactive Information Retrieval Tasks

Authors:

Daniel Hienert,

Matthew Mitsui,

Nicholas J. BelkinAuthors Info & Claims

CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

Pages 73 - 81

https://doi.org/10.1145/3295750.3298921

Published: 08 March 2019 Publication History

Abstract

In Interactive Information Retrieval (IIR) experiments the user's gaze motion on web pages is often recorded with eye tracking. The data is used to analyze gaze behavior or to identify Areas of Interest (AOI) the user has looked at. So far, tools for analyzing eye tracking data have certain limitations in supporting the analysis of gaze behavior in IIR experiments. Experiments often consist of a huge number of different visited web pages. In existing analysis tools the data can only be analyzed in videos or images and AOIs for every single web page have to be specified by hand, in a very time consuming process. In this work, we propose the reading protocol software which breaks eye tracking data down to the textual level by considering the HTML structure of the web pages. This has a lot of advantages for the analyst. First and foremost, it can easily be identified on a large scale what has actually been viewed and read on the stimuli pages by the subjects. Second, the web page structure can be used to filter to AOIs. Third, gaze data of multiple users can be presented on the same page, and fourth, fixation times on text can be exported and further processed in other tools. We present the software, its validation, and example use cases with data from three existing IIR experiments.

References

[1]

Ajanki, A., Hardoon, D.R., Kaski, S., Puolamäki, K. and Shawe-Taylor, J. 2009. Can eyes reveal interest? Implicit queries from gaze patterns. User Modeling and User-Adapted Interaction. 19, 4 (Oct. 2009), 307--339.

Digital Library

[2]

Balatsoukas, P. and Ruthven, I. 2012. An Eye-tracking Approach to the Analysis of Relevance Judgments on the Web: The Case of Google Search Engine. J. Am. Soc. Inf. Sci. Technol. 63, 9 (Sep. 2012), 1728--1746.

Digital Library

[3]

Belkin, N.J. 1980. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science. 5 (1980), 133--143.

[4]

Belkin, N.J., Cool, C., Stein, A. and Thiel, U. 1995. Cases, Scripts, and Information-Seeking Strategies: On the Design of Interactive Information Retrieval Systems. EXPERT SYSTEMS WITH APPLICATIONS. 9, (1995), 379--395.

[5]

Beymer, D. and Russell, D.M. 2005. WebGazeAnalyzer: A System for Capturing and Analyzing Web Reading Behavior Using Eye Gaze. CHI '05 Extended Abstracts on Human Factors in Computing Systems (New York, NY, USA, 2005), 1913--1916.

Digital Library

[6]

Biedert, R., Buscher, G. and Dengel, A. 2010. The eyeBook -- Using Eye Tracking to Enhance the Reading Experience. Informatik-Spektrum. 33, 3 (2010), 272--281.

[7]

Blascheck, T., Kurzhals, K., Raschke, M., Burch, M., Weiskopf, D. and Ertl, T. Visualization of Eye Tracking Data: A Taxonomy and Survey. Computer Graphics Forum. 36, 8, 260--284.

[8]

Blignaut, P., Holmqvist, K., Nyström, M. and Dewhurst, R. 2014. Improving the Accuracy of Video-Based Eye Tracking in Real Time through Post-Calibration Regression. Current Trends in Eye Tracking Research. M. Horsley, M. Eliot, B.A. Knight, and R. Reilly, eds. Springer International Publishing. 77--100.

[9]

Borlund, P. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research: an international electronic journal. (2003).

[10]

Buscher, G., Dengel, A. and van Elst, L. 2008. Query Expansion Using Gaze-based Feedback on the Subdocument Level. Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New York, NY, USA, 2008), 387--394.

Digital Library

[11]

Carevic, Z., Lusky, M., Hoek, W. van and Mayr, P. 2017. Investigating exploratory search activities based on the stratagem level in digital libraries. International Journal on Digital Libraries. (2017), 1--21.

Digital Library

[12]

Cole, M., Liu, J., Belkin, N., Bierig, R., Gwizdka, J., Liu, C., Zhang, J. and Zhang, X. 2009. Usefulness as the criterion for evaluation of interactive information retrieval. Proceedings of the Workshop on Human-Computer Interaction and Information Retrieval (2009), 1--4.

[13]

Cole, M.J., Gwizdka, J., Liu, C., Belkin, N.J. and Zhang, X. 2013. Inferring User Knowledge Level from Eye Movement Patterns. Inf. Process. Manage. 49, 5 (Sep. 2013), 1075--1091.

Digital Library

[14]

Cole, M.J., Gwizdka, J., Liu, C., Bierig, R., Belkin, N.J. and Zhang, X. 2011. Task and user effects on reading patterns in information search. Interacting with Computers. 23, 4 (2011), 346--362.

[15]

Cole, M.J., Hendahewa, C., Belkin, N.J. and Shah, C. 2014. Discrimination Between Tasks with User Activity Patterns During Information Search. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (New York, NY, USA, 2014), 567--576.

Digital Library

[16]

Eickhoff, C., Dungs, S. and Tran, V. 2015. An Eye-Tracking Study of Query Reformulation. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (New York, NY, USA, 2015), 13--22.

Digital Library

[17]

Granka, L.A., Joachims, T. and Gay, G. 2004. Eye-tracking Analysis of User Behavior in WWW Search. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New York, NY, USA, 2004), 478--479.

Digital Library

[18]

Gwizdka, J. 2014. Characterizing Relevance with Eye-tracking Measures. Proceedings of the 5th Information Interaction in Context Symposium (New York, NY, USA, 2014), 58--67.

Digital Library

[19]

Hienert, D., Mitsui, M., Mayr, P., Shah, C. and Belkin, N.J. 2018. The Role of the Task Topic in Web Search of Different Task Types. Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (New York, NY, USA, 2018), 72--81.

Digital Library

[20]

Hienert, D., Sawitzki, F. and Mayr, P. 2015. Digital Library Research in Action - Supporting Information Retrieval in Sowiport. D-Lib Magazine. 21, 3/4 (2015).

[21]

Ingwersen, P. and Järvelin, K. 2007. On the Holistic Cognitive Theory for Information Retrieval: Drifting Outside the Border of the Laboratory Framework. Studies in the Theory of Information Retrieval (ICTIR 2007). Budapest, Hungary: Foundation for Information Society (2007), 135--147.

[22]

Jarodzka, H. and Brand-Gruwel, S. Tracking the reading eye: towards a model of real-world reading. Journal of Computer Assisted Learning. 33, 3, 193--201.

[23]

Jiang, J., He, D. and Allan, J. 2014. Searching, Browsing, and Clicking in a Search Session: Changes in User Behavior by Task and over Time. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (New York, NY, USA, 2014), 607--616.

Digital Library

[24]

Kellar, M., Watters, C. and Shepherd, M. 2007. A Field Study Characterizing Web-based Information-seeking Tasks. J. Am. Soc. Inf. Sci. Technol. 58, 7 (May 2007), 999--1018.

Digital Library

[25]

Kern, D., Hienert, D., Angerbauer, K., Tilman, D. and Borlund, P. Lessons Learned from Users Reading Highlighted Abstracts in a Digital Library. CHIIR '19 Proceedings of the 2019 Conference on Human Information Interaction & Retrieval.

Digital Library

[26]

Li, Y. and Belkin, N.J. 2008. A faceted approach to conceptualizing tasks in information seeking. Information Processing & Management. 44, 6 (2008), 1822--1837.

Digital Library

[27]

Reeder, R.W., Pirolli, P. and Card, S.K. 2001. WebEyeMapper and WebLogger: Tools for Analyzing Eye Tracking Data Collected in Web-use Studies. CHI '01 Extended Abstracts on Human Factors in Computing Systems (New York, NY, USA, 2001), 19--20.

Digital Library

[28]

Reichle, E.D., Pollatsek, A. and Rayner, K. 2006. E-Z Reader: A Cognitive-control, Serial-attention Model of Eye-movement Behavior During Reading. Cogn. Syst. Res. 7, 1 (Mar. 2006), 4--22.

Digital Library

[29]

Reichle, E.D., Pollatsek, A. and Rayner, K. 2012. Using E-Z Reader to simulate eye movements in nonreading tasks: a unified framework for understanding the eye-mind link. Psychological review. 119, 1 (Jan. 2012), 155--185.

[30]

Salvucci, D.D. and Goldberg, J.H. 2000. Identifying Fixations and Saccades in Eye-tracking Protocols. Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (New York, NY, USA, 2000), 71--78.

Digital Library

[31]

Spakov, O., Siirtola, H., Istance, H. and Räihä, K.-J. 2017. Visualizing the Reading Activity of People Learning to Read. Journal of Eye Movement Research. 10, 5 (2017).

[32]

Tang, S., Reilly, R.G. and Vorstius, C. 2012. EyeMap: a software system for visualizing and analyzing eye movement data in reading. Behavior Research Methods. 44, 2 (Jun. 2012), 420--438.

[33]

Umemoto, K., Yamamoto, T., Nakamura, S. and Tanaka, K. 2012. Search Intent Estimation from User's Eye Movements for Supporting Information Seeking. Proceedings of the International Working Conference on Advanced Visual Interfaces (New York, NY, USA, 2012), 349--356.

Digital Library

[34]

Vakkari, P. 2016. Searching as learning: A systematization based on literature. Journal of Information Science. 42, 1 (2016), 7--18.

Digital Library

[35]

Vakkari, P. 2003. Task-based information searching. Annual Review of Information Science and Technology. 37, 1 (2003), 413--464.

Cited By

Hienert DSchmidt HKrämer TKern D(2024)EyeLiveMetrics: Real-time Analysis of Online Reading with Eye TrackingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656495(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3656495
Nasser HEl Zein Dda Costa Pereira CEscazut CTettamanzi A(2024)RULKKG: Estimating User’s Knowledge Gain in Search-as-Learning Using Knowledge GraphsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638331(364-369)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638331
Gritz WHoppe AEwerth R(2024)On the Influence of Reading Sequences on Knowledge Gain During Web SearchAdvances in Information Retrieval10.1007/978-3-031-56063-7_28(364-373)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56063-7_28
Show More Cited By

Index Terms

Reading Protocol: Understanding what has been Read in Interactive Information Retrieval Tasks
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Quantifying reading habits: counting how many words you read
UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Reading is a very common learning activity, a lot of people perform it everyday even while standing in the subway or waiting in the doctors office. However, we know little about our everyday reading habits, quantifying them enables us to get more ...
Eye movements during reading and reading assessment in swedish school children: a new window on reading difficulties
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Research during the last decades has demonstrated that eye tracking methodology is an advantageous tool to study reading. A substantial amount of eye movement research has resulted in improved understanding of the reading process in skilled adult ...
Analysing EOG signal features for the discrimination of eye movements with wearable devices
PETMEI '11: Proceedings of the 1st international workshop on pervasive eye tracking & mobile eye-based interaction

Eye tracking research in human-computer interaction and experimental psychology traditionally focuses on stationary devices and a small number of common eye movements. The advent of pervasive eye tracking promises new applications, such as eye-based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

March 2019

463 pages

ISBN:9781450360258

DOI:10.1145/3295750

General Chairs:
Leif Azzopardi
University of Strathclyde, UK
,
Martin Halvey
University of Strathclyde, UK
,
Ian Ruthven
University of Strathclyde, UK
,
Program Chairs:
Hideo Joho
University of Tsukuba, Japan
,
Vanessa Murdock
Amazon, USA
,
Pernilla Qvarfordt
pxq Labs, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CHIIR '19

Sponsor:

CHIIR '19: Conference on Human Information Interaction and Retrieval

March 10 - 14, 2019

Glasgow, Scotland UK

Acceptance Rates

Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
524
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)7

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hienert DSchmidt HKrämer TKern D(2024)EyeLiveMetrics: Real-time Analysis of Online Reading with Eye TrackingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656495(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3656495
Nasser HEl Zein Dda Costa Pereira CEscazut CTettamanzi A(2024)RULKKG: Estimating User’s Knowledge Gain in Search-as-Learning Using Knowledge GraphsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638331(364-369)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638331
Gritz WHoppe AEwerth R(2024)On the Influence of Reading Sequences on Knowledge Gain During Web SearchAdvances in Information Retrieval10.1007/978-3-031-56063-7_28(364-373)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56063-7_28
Barz MBhatti OSonntag D(2022)Implicit Estimation of Paragraph Relevance From Eye MovementsFrontiers in Computer Science10.3389/fcomp.2021.8085073Online publication date: 7-Jan-2022
https://doi.org/10.3389/fcomp.2021.808507
Valdunciel PBhatti OBarz MSonntag DElsweiler DKruschwitz ULudwig B(2022)Interactive Assessment Tool for Gaze-based Machine Learning Models in Information RetrievalProceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505834(332-336)Online publication date: 14-Mar-2022
https://dl.acm.org/doi/10.1145/3498366.3505834
Pardi GHienert DKammerer Y(2022)Examining the use of text and video resources during web-search based learning—a new methodological approachNew Review of Hypermedia and Multimedia10.1080/13614568.2022.209958328:1-2(39-67)Online publication date: 14-Jul-2022
https://doi.org/10.1080/13614568.2022.2099583
Wallace SPapoutsaki ATan NGuo HHuang J(2021)Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular DatasetsProceedings of the ACM on Human-Computer Interaction10.1145/34795925:CSCW2(1-25)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479592
Schmidt SZlatkin-Troitschanskaia ORoeper JKlose VWeber MBültmann ABrückner S(2020)Undergraduate Students’ Critical Online Reasoning—Process Mining AnalysisFrontiers in Psychology10.3389/fpsyg.2020.57627311Online publication date: 30-Nov-2020
https://doi.org/10.3389/fpsyg.2020.576273

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents