demonstration

SaL-Lightning Dataset: Search and Eye Gaze Behavior, Resource Interactions and Knowledge Gain during Web Search

Authors:

Christian Otto,

Markus Rokicki,

Wolfgang Gritz,

Daniel Hienert,

Johannes von Hoyer,

Yvonne Kammerer,

Ralph EwerthAuthors Info & Claims

CHIIR '22: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

Pages 347 - 352

https://doi.org/10.1145/3498366.3505835

Published: 14 March 2022 Publication History

Abstract

The emerging research field Search as Learning (SAL) investigates how the Web facilitates learning through modern information retrieval systems. SAL research requires significant amounts of data that capture both search behavior of users and their acquired knowledge in order to obtain conclusive insights or train supervised machine learning models. However, the creation of such datasets is costly and requires interdisciplinary efforts in order to design studies and capture a wide range of features. In this paper, we address this issue and introduce an extensive dataset based on a user study, in which 114 participants were asked to learn about the formation of lightning and thunder. Participants’ knowledge states were measured before and after Web search through multiple-choice questionnaires and essay-based free recall tasks. To enable future research in SAL-related tasks we recorded a plethora of features and person-related attributes. Besides the screen recordings, visited Web pages, and detailed browsing histories, a large number of behavioral features and resource features were monitored. We underline the usefulness of the dataset by describing three, already published, use cases.

References

[1]

Khalid Al-Khatib, Michael Völske, Shahbaz Syed, Nikolay Kolyada, and Benno Stein. 2020. Webis ChangeMyView Corpus 2020 (Webis-CMV-20). https://doi.org/10.5281/zenodo.3778298

[2]

Lorin W Anderson, David R Krathwohl, P Airasian, K Cruikshank, R Mayer, P Pintrich, J Raths, and M Wittrock. 2001. A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy. Educational Horizons 9, 2 (2001), 137–175.

[3]

Ivar Bråten, Christian Brandmo, and Yvonne Kammerer. 2019. A Validation Study of the Internet-Specific Epistemic Justification Inventory With Norwegian Preservice Teachers. Journal of Educational Computing Research 57, 4 (2019), 877–900. https://doi.org/10.1177/0735633118769438 arXiv:https://doi.org/10.1177/0735633118769438

[4]

Romain Cohendet, Claire-Hélène Demarty, Ngoc Q. K. Duong, and Martin Engilberge. 2019. VideoMem: Constructing, Analyzing, Predicting Short-Term and Long-Term Video Memorability. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 2531–2540. https://doi.org/10.1109/ICCV.2019.00262

[5]

Andrew R. A. Conway, Michael J. Kane, Michael F. Bunting, D. Zach Hambrick, Oliver Wilhelm, and Randall W. Engle. 2005. Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review 12, 5 (2005), 769–786. https://doi.org/10.3758/BF03196772

[6]

Stefan Engeser. 2005. Messung des expliziten Leistungsmotivs: Kurzform der Achievement Motives Scale. (2005). https://www.uni-trier.de/fileadmin/fb1/prof/PSY/PGA/bilder/Engeser__2005__Kurzform_der_AMS.pdf

[7]

Shane Frederick. 2005. Cognitive Reflection and Decision Making. Journal of Economic Perspectives 19, 4 (December 2005), 25–42. https://doi.org/10.1257/089533005775196732

[8]

Ujwal Gadiraju, Ran Yu, Stefan Dietze, and Peter Holtz. 2018. Analyzing Knowledge Gain of Users in Informational Search Sessions on the Web. In Proceedings of the 2018 Conference on Human Information Interaction and Retrieval, CHIIR 2018, New Brunswick, NJ, USA, March 11-15, 2018, Chirag Shah, Nicholas J. Belkin, Katriina Byström, Jeff Huang, and Falk Scholer (Eds.). ACM, 2–11.

Digital Library

[9]

Raphaël Grolimund. 2017. Individual work of Bachelor students monitored using a dynamic assessment dashboard. https://doi.org/10.5281/zenodo.290129

[10]

Matthias Hagen, Martin Potthast, Benno Stein, Christof Bräutigam, and Anna Beyer. 2010. Webis Query Segmentation Corpus 2010 (Webis- QSeC-10). https://doi.org/10.5281/zenodo.3256198

[11]

Matthias Hagen, Martin Potthast, Benno Stein, Marcel Gohsen, and Anja Rathgeber. 2017. Webis Query Spelling Corpus 2017 (Webis-QSpell-17). https://doi.org/10.5281/zenodo.3570912

[12]

Matthias Hagen, Benno Stein, Jakob Gomoll, and Anna Beyer. 2013. Webis Search Mission Corpus 2012 (Webis-SMC-12). https://doi.org/10.5281/zenodo.3265962

[13]

Avery Hlousek and Bentley Krause. 2020. The impact of learning modality on team-based learning (TBL) outcomes in anatomical sciences education. https://doi.org/10.5061/dryad.cfxpnvx35

[14]

Phillip Isola, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2011. What makes an image memorable?. In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011. IEEE Computer Society, 145–152. https://doi.org/10.1109/CVPR.2011.5995721

Digital Library

[15]

Gerald Matthews, Sian E Campbell, Shona Falconer, Lucy A Joyner, Jane Huggins, Kirby Gilliland, Rebecca Grier, and Joel S Warm. 2002. Fundamental dimensions of subjective state in performance settings: task engagement, distress, and worry. Emotion (Washington, D.C.) 2, 4 (December 2002), 315—340. https://doi.org/10.1037/1528-3542.2.4.315

[16]

Richard E Mayer and Roxana Moreno. 1998. A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of educational psychology 90, 2 (1998), 312.

[17]

Christian Otto, Ran Yu, Georg Pardi, Johannes von Hoyer, Markus Rokicki, Anett Hoppe, Peter Holtz, Yvonne Kammerer, Stefan Dietze, and Ralph Ewerth. 2021. Predicting Knowledge Gain During Web Search Based on Multimedia Resource Consumption. In Artificial Intelligence in Education - 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14-18, 2021, Proceedings, Part I(Lecture Notes in Computer Science, Vol. 12748), Ido Roll, Danielle S. McNamara, Sergey A. Sosnovsky, Rose Luckin, and Vania Dimitrova (Eds.). Springer, 318–330. https://doi.org/10.1007/978-3-030-78292-4_26

Digital Library

[18]

Georg Pardi, Johannes von Hoyer, Peter Holtz, and Yvonne Kammerer. 2020. The Role of Cognitive Abilities and Time Spent on Texts and Videos in a Multimodal Searching as Learning Task. In CHIIR ’20: Conference on Human Information Interaction and Retrieval, Vancouver, BC, Canada, March 14-18, 2020, Heather L. O’Brien, Luanne Freund, Ioannis Arapakis, Orland Hoeber, and Irene Lopatovska (Eds.). ACM, 378–382. https://doi.org/10.1145/3343413.3378001

Digital Library

[19]

Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset. CoRR abs/1912.04639(2019). arxiv:1912.04639http://arxiv.org/abs/1912.04639

[20]

Verónica Proaño-Ríosand Roberto I. González-Ibáñez. 2020. Dataset of Search Results Organized as Learning Paths Recommended by Experts to Support Search as Learning. Data 5, 4 (2020), 92. https://doi.org/10.3390/data5040092

[21]

Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the Eye Tracking Research & Application Symposium, ETRA 2000, Palm Beach Gardens, Florida, USA, November 6-8, 2000, Andrew T. Duchowski (Ed.). ACM, 71–78. https://doi.org/10.1145/355017.355028

Digital Library

[22]

Florian Schmidt-Weigand and Katharina Scheiter. 2011. The role of spatial descriptions in learning from multimedia. Comput. Hum. Behav. 27, 1 (2011), 22–28.

Digital Library

[23]

Wolfgang Schneider, Matthias Schlagmüller, and Marco Ennemoser. 2007. LGVT 6-12: Lesegeschwindigkeits-und-verständnistest für die Klassen 6-12. Hogrefe Göttingen.

[24]

Davide Taibi, Francesca Bianchi, Philipp Kemkes, and Ivana Marenzi. 2018. Learning Analytics for Interpreting. In Proceedings of the 10th International Conference on Computer Supported Education, CSEDU 2018, Funchal, Madeira, Portugal, March 15-17, 2018, Volume 1, Bruce M. McLaren, Rob Reilly, Susan Zvacek, and James Onohuome Uhomoibhi (Eds.). SciTePress, 145–154. https://doi.org/10.5220/0006774801450154

[25]

Katrien Verbert, Hendrik Drachsler, Nikos Manouselis, Martin Wolpers, Riina Vuorikari, and Erik Duval. 2011. Dataset-driven research for improving recommender systems for learning. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, LAK 2011, Banff, AB, Canada, February 27 - March 01, 2011, Phillip Long, George Siemens, Gráinne Conole, and Dragan Gasevic (Eds.). ACM, 44–53. https://doi.org/10.1145/2090116.2090122

Digital Library

[26]

Michael Völske, Matthias Hagen, and Benno Stein. 2019. Webis Query-Task-Mapping Corpus 2019 (Webis- QTM-19). https://doi.org/10.5281/zenodo.3257431

[27]

Johannes von Hoyer, Georg Pardi, Yvonne Kammerer, and Peter Holtz. 2019. Metacognitive Judgments in Searching as Learning (SAL) Tasks: Insights on (Mis-) Calibration, Multimedia Usage, and Confidence. In Proceedings of the 1st International Workshop on Search as Learning with Multimedia Information(Nice, France) (SALMM ’19). Association for Computing Machinery, New York, NY, USA, 3–10. https://doi.org/10.1145/3347451.3356730

Digital Library

[28]

Mark Warschauer, Lynn Reimer, Kameryn Denaro, Gabe Orona, Katerina Schenke, Tutrang Nguyen, Amanda Niili, Di Xu, Sabrina Solanki, and Tate Tamara. 2021. Evaluating Promising Practices in Undergraduate STEM Lecture Courses. https://doi.org/10.7280/D11M5Q

Cited By

Nasser HEl Zein Dda Costa Pereira CEscazut CTettamanzi A(2024)RULKKG: Estimating User’s Knowledge Gain in Search-as-Learning Using Knowledge GraphsProceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval10.1145/3627508.3638331(364-369)Online publication date: 10-Mar-2024
https://doi.org/10.1145/3627508.3638331
Dietze SChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Archiving and Temporal Analysis of Behavioral Web Data - Tales from the InsideCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3641260(1373-1374)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641260

Index Terms

SaL-Lightning Dataset: Search and Eye Gaze Behavior, Resource Interactions and Knowledge Gain during Web Search

Index terms have been assigned to the content through auto-classification.

Recommendations

Analyzing Knowledge Gain of Users in Informational Search Sessions on the Web
CHIIR '18: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval

Web search is frequently used by people to acquire new knowledge and to satisfy learning-related objectives, but little is known about how a user»s knowledge evolves through the course of a search session. We present a study addressing the knowledge ...
Predicting Knowledge Gain During Web Search Based on Multimedia Resource Consumption
Artificial Intelligence in Education
Abstract
In informal learning scenarios the popularity of multimedia content, such as video tutorials or lectures, has significantly increased. Yet, the users’ interactions, navigation behavior, and consequently learning outcome, have not been researched ...
On the Influence of Reading Sequences on Knowledge Gain During Web Search
Advances in Information Retrieval
Abstract
Nowadays, learning increasingly involves the usage of search engines and web resources. The related interdisciplinary research field search as learning aims to understand how people learn on the web. Previous work has investigated several feature ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHIIR '22: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

March 2022

399 pages

ISBN:9781450391863

DOI:10.1145/3498366

General Chairs:
David Elsweiler
University of Regensburg, Bavaria, Germany
,
Udo Kruschwitz
University of Regensburg, Bavaria, Germany
,
Bernd Ludwig
University of Regensburg, Bavaria, Germany

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2022

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

CHIIR '22

Sponsor:

CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval

March 14 - 18, 2022

Regensburg, Germany

Acceptance Rates

Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
163
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)3

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nasser HEl Zein Dda Costa Pereira CEscazut CTettamanzi A(2024)RULKKG: Estimating User’s Knowledge Gain in Search-as-Learning Using Knowledge GraphsProceedings of the 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval10.1145/3627508.3638331(364-369)Online publication date: 10-Mar-2024
https://doi.org/10.1145/3627508.3638331
Dietze SChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Archiving and Temporal Analysis of Behavioral Web Data - Tales from the InsideCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3641260(1373-1374)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641260

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents