research-article

Open access

Pausing while programming: insights from keystroke analysis

Authors:

Albina Zavgorodniaia,

John EdwardsAuthors Info & Claims

ICSE-SEET '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training

Pages 187 - 198

https://doi.org/10.1145/3510456.3514146

Published: 17 October 2022 Publication History

Abstract

Pauses in typing are generally considered to indicate cognitive processing and so are of interest in educational contexts. While much prior work has looked at typing behavior of Computer Science students, this paper presents results of a study specifically on the pausing behavior of students in Introductory Computer Programming. We investigate the frequency of pauses of different lengths, what last actions students take before pausing, and whether there is a correlation between pause length and performance in the course. We find evidence that frequency of pauses of all lengths is negatively correlated with performance, and that, while some keystrokes initiate pauses consistently across pause lengths, other keystrokes more commonly initiate short or long pauses. Clustering analysis discovers two groups of students, one that takes relatively fewer mid-to-long pauses and performs better on exams than the other.

References

[1]

Alireza Ahadi, Raymond Lister, Heikki Haapala, and Arto Vihavainen. 2015. Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the eleventh annual international conference on international computing education research. 121--130.

Digital Library

[2]

Amjad Altadmri and Neil C.C. Brown. 2015. 37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE '15). Association for Computing Machinery, New York, NY, USA, 522--527.

Digital Library

[3]

Amjad Altadmri, Michael Kolling, and Neil CC Brown. 2016. The cost of syntax and how to avoid it: Text versus frame-based editing. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC). IEEE, 748--753.

[4]

Erik M Altmann and J Gregory Trafton. 2007. Timecourse of recovery from task interruption: Data and a model. Psychonomic Bulletin & Review 14, 6 (2007), 1079--1084.

[5]

Rui A Alves, São Luís Castro, Liliana de Sousa, and Sven Strömqvist. 2007. Chapter 4: Influence of Typing Skill on Pause-Execution Cycles in Written Composition. In Writing and Cognition. BRILL, 55--65.

[6]

Mark B. Edwards and Scott D Gronlund. 1998. Task interruption and its effects on memory. Memory 6, 6 (1998), 665--687.

[7]

Jens Bennedsen and Michael E Caspersen. 2006. Abstraction ability as an indicator of success for learning object-oriented programming? ACM Sigcse Bulletin 38, 2 (2006), 39--43.

Digital Library

[8]

Susan Bergin and Ronan Reilly. 2005. Programming: factors that influence success. In Proceedings of the 36th SIGCSE technical symposium on Computer science education. 411--415.

Digital Library

[9]

Jelmer P Borst, Niels A Taatgen, and Hedderik van Rijn. 2015. What makes interruptions disruptive?: A process-model account of the effects of the problem state bottleneck on task interruption and resumption. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, 2971--2980.

Digital Library

[10]

Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: a large scale repository of novice programmers' activity. In Proceedings of the 45th ACM technical symposium on Computer science education. ACM, 223--228.

Digital Library

[11]

Brian L. Butterworth. 1980. Evidence from pauses in speech. New York: Academic Press.

[12]

Adam S Carter, Christopher D Hundhausen, and Olusola Adesope. 2015. The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the eleventh annual international conference on international computing education research. 141--150.

Digital Library

[13]

Jasone Cenoz. 2000. Pauses and hesitation phenomena in second language production. ITL - International Journal of Applied Linguistics 127-128 (Jan. 2000), 53--69.

[14]

Markus F Damian and Hans Stadthagen-Gonzalez. 2009. Advance planning of form properties in the written production of single and multiple words. Language and Cognitive Processes 24, 4 (2009), 555--579.

[15]

Simon P Davies. 1991. The role of notation and knowledge representation in the determination of programming strategy: a framework for integrating models of programming behavior. Cognitive Science 15, 4 (1991), 547--572.

[16]

Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All Syntax Errors Are Not Equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (Haifa, Israel) (ITiCSE '12). ACM, New York, NY, USA, 75--80.

Digital Library

[17]

John Edwards, Joseph Ditton, Dragan Trninic, Hillary Swanson, Shelsey Sullivan, and Chad Mano. 2020. Syntax exercises in CS1. In Proceedings of the 16th Annual Conference on International Computing Education Research (Dunedin, New Zealand) (ICER '20).

Digital Library

[18]

John Edwards, Juho Leinonen, Chetan Birthare, Albina Zavgorodniaia, and Arto Hellas. 2020. Programming Versus Natural Language: On the Effect of Context on Typing in CS1. In Proceedings of the 2020 ACM Conference on International Computing Education Research. 204--215.

Digital Library

[19]

John Edwards, Juho Leinonen, and Arto Hellas. 2020. A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 413--419.

Digital Library

[20]

Clayton Epp, Michael Lippold, and Regan L Mandryk. 2011. Identifying emotional states using keystroke dynamics. In Proceedings of the sigchi conference on human factors in computing systems. 715--724.

Digital Library

[21]

Jean-Noël Foulin. 1995. Pauses et débits : les indicateurs temporels de la production écrite. L'année psychologique 95, 3 (1995), 483--504.

[22]

Tony Gillie and Donald Broadbent. 1989. What makes interruptions disruptive? A study of length, similarity, and complexity. Psychological research 50, 4 (1989), 243--250.

[23]

Alexander JJ Gould. 2014. What makes an interruption disruptive? Understanding the effects of interruption relevance and timing on performance. Ph.D. Dissertation. UCL (University College London).

[24]

Winston Haynes. 2013. Bonferroni Correction. Springer New York, New York, NY, 154--154.

[25]

Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018. Predicting academic performance: a systematic literature review. In Proceedings companion of the 23rd annual ACM conference on innovation and technology in computer science education. 175--199.

Digital Library

[26]

Arto Hellas, Juho Leinonen, and Petri Ihantola. 2017. Plagiarism in take-home exams: Help-seeking, collaboration, and systematic cheating. In Proceedings of the 2017 ACM conference on innovation and technology in computer science education. 238--243.

Digital Library

[27]

C. D. Hundhausen, D. M. Olivares, and A. S. Carter. 2017. IDE-Based Learning Analytics for Computing Education: A Process Model, Critical Review, and Research Agenda. ACM Trans. Comput. Educ. 17, 3, Article 11 (Aug. 2017), 26 pages.

Digital Library

[28]

Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proc. of the 2015 ITiCSE on Working Group Reports (Vilnius, Lithuania) (ITICSE-WGR '15). ACM, 41--63.

Digital Library

[29]

Shamsi T Iqbal and Brian P Bailey. 2006. Leveraging characteristics of task structure to predict the cost of interruption. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 741--750.

Digital Library

[30]

Matthew C Jadud. 2005. A first look at novice compilation behaviour using BlueJ. Computer Science Education 15, 1 (2005), 25--40.

[31]

Matthew C Jadud. 2006. Methods and tools for exploring novice compilation behaviour. In Proceedings of the second international workshop on Computing education research. ACM, 73--84.

Digital Library

[32]

Agata Kołakowska. 2016. Towards detecting programmers' stress on the basis of keystroke dynamics. In 2016 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, 1621--1626.

[33]

Minna Kumpulainen. 2015. On the operationalisation of 'pauses' in translation process research. Translation & Interpreting 7, 1 (2015), 47--58.

[34]

Isabel Lacruz and Gregory M. Shreve. 2014. Pauses and Cognitive Effort in Post-Editing. In Post-editing of Machine Translation: Processes and Applications. Cambridge Scholars Publishing.

[35]

Joy Yeonjoo Lee, Jeroen Donkers, Halszka Jarodzka, Géraldine Sellenraad, and Jeroen J.G. van Merriënboer. 2020. Different effects of pausing on cognitive load in a medical simulation game. Computers in Human Behavior 110 (Sept. 2020), 106385.

[36]

Marianne Leinikka, Arto Vihavainen, Jani Lukander, and Satu Pakarinen. 2014. Cognitive flexibility and programming performance. In Psychology of programming interest group workshop. 1--11.

[37]

Juho Leinonen. 2019. Keystroke Data in Programming Courses. Ph.D. Dissertation. University of Helsinki.

[38]

Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas. 2021. Fine-Grained Versus Coarse-Grained Data for Estimating Time-on-Task in Learning Programming. In Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021). The International Educational Data Mining Society.

[39]

Juho Leinonen, Leo Leppänen, Petri Ihantola, and Arto Hellas. 2017. Comparison of time metrics in programming. In Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM, 200--208.

Digital Library

[40]

Juho Leinonen, Krista Longi, Arto Klami, and Arto Vihavainen. 2016. Automatic inference of programming performance and experience from typing patterns. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 132--137.

Digital Library

[41]

Leo Leppänen, Juho Leinonen, and Arto Hellas. 2016. Pauses and spacing in learning to program. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research. ACM, 41--50.

Digital Library

[42]

Soohyun Nam Liao, Daniel Zingaro, Kevin Thai, Christine Alvarado, William G. Griswold, and Leo Porter. 2019. A Robust Machine Learning Technique to Predict Low-Performing Students. ACM Trans. Comput. Educ. 19, 3, Article 18 (Jan. 2019), 19 pages.

Digital Library

[43]

Sharon O'Brien. 2006. Pauses as Indicators of Cognitive Effort in Post-editing Machine Translation Output. Across Languages and Cultures 7, 1 (June 2006), 1--21.

[44]

Thierry Olive, Rui Alexandre Alves, and São Luís Castro. 2009. Cognitive processes in writing during pause and execution periods. European Journal of Cognitive Psychology 21, 5 (Aug. 2009), 758--785.

[45]

David N Perkins, Chris Hancock, Renee Hobbs, Fay Martin, and Rebecca Simmons. 1986. Conditions of learning in novice programmers. Journal of Educational Computing Research 2, 1 (1986), 37--55.

[46]

Andrew Petersen, Jaime Spacco, and Arto Vihavainen. 2015. An exploration of error quotient in multiple contexts. In Proceedings of the 15th Koli Calling Conference on Computing Education Research. 77--86.

Digital Library

[47]

Leo Porter, Daniel Zingaro, and Raymond Lister. 2014. Predicting Student Success Using Fine Grain Clicker Data. In Proceedings of the Tenth Annual Conference on International Computing Education Research (Glasgow, Scotland, United Kingdom) (ICER '14). Association for Computing Machinery, New York, NY, USA, 51--58.

Digital Library

[48]

Andrea Révész, Marije Michel, and MinJin Lee. 2017. Investigating IELTS Academic Writing Task 2: Relationships between cognitive writing processes, text quality, and working memory. British Council, Cambridge English Language Assessment and IDP.

[49]

Andrea Révész, Marije Michel, and Minjin Lee. 2019. EXPLORING SECOND LANGUAGE WRITERS' PAUSING AND REVISION BEHAVIORS. Studies in Second Language Acquisition 41, 3 (July 2019), 605--631.

[50]

Russell Revlin. 2013. Cognition : theory and practice. Worth Publishers, New York, NY.

[51]

Robert S Rist. 1989. Schema creation in programming. Cognitive Science 13, 3 (1989), 389--414.

[52]

Robert S Rist. 1995. Program structure and design. Cognitive science 19, 4 (1995), 507--562.

[53]

Nathan Rountree, Janet Rountree, Anthony Robins, and Robert Hannah. 2004. Interacting factors that predict success and failure in a CS1 course. ACM SIGCSE Bulletin 36, 4 (2004), 101--104.

Digital Library

[54]

Joost Schilperoord. 1996. It's about time: Temporal aspects of cognitive processes in text production. Vol. 6. Rodopi.

[55]

Richard C Thomas, Amela Karahasanovic, and Gregor E Kennedy. 2005. An investigation into keystroke latency metrics as an indicator of programming performance. In Proceedings of the 7th Australasian conference on Computing education-Volume 42. 127--134.

Digital Library

[56]

Robert L Thorndike. 1953. Who belongs in the family? Psychometrika 18, 4 (1953), 267--276.

[57]

Markku Tukiainen and Eero Mönkkönen. 2002. Programming Aptitude Testing as a Prediction of Learning to Program. In PPIG. 4.

[58]

Jeroen JG Van Merrienboer and Fred GWC Paas. 1990. Automation and schema acquisition in learning elementary computer programming: Implications for the design of practice. Computers in Human Behavior 6, 3 (1990), 273--289.

[59]

Arto Vihavainen, Juha Helminen, and Petri Ihantola. 2014. How Novices Tackle Their First Lines of Code in an IDE: Analysis of Programming Session Traces. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '14). ACM, New York, NY, USA, 109--116.

Digital Library

[60]

Arto Vihavainen, Matti Luukkainen, and Petri Ihantola. 2014. Analysis of source code snapshot granularity levels. In Proceedings of the 15th annual conference on information technology education. 21--26.

Digital Library

[61]

Arto Vihavainen, Thomas Vikberg, Matti Luukkainen, and Martin Pärtel. 2013. Scaffolding students' learning using test my code. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 117--122.

Digital Library

[62]

Luuk Van Waes and Peter Jan Schellens. 2003. Writing profiles: the effect of the writing mode on pausing and revision patterns of experienced writers. Journal of Pragmatics 35, 6 (June 2003), 829--853.

[63]

Ronald L Wasserstein and Nicole A Lazar. 2016. The ASA statement on p-values: context, process, and purpose.

[64]

Christopher Watson, Frederick WB Li, and Jamie L Godwin. 2013. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In 2013 IEEE 13th international conference on advanced learning technologies. IEEE, 319--323.

Digital Library

[65]

Laurie Honour Werth. 1986. Predicting student performance in a beginning computer science class. ACM SIGCSE Bulletin 18, 1 (1986), 138--143.

Digital Library

[66]

Leon E Winslow. 1996. Programming pedagogy---a psychological overview. ACM Sigcse Bulletin 28, 3 (1996), 17--22.

Digital Library

Cited By

Karakaš AHelic DMonga MLonati VBarendsen ESheard JPaterson J(2024)Combining Local Testing with Automatic Commits: Benefits for Progress Tracking and CS2 Students' Learning ExperienceProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653561(108-114)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653561
Brown NGuzdial MStephenson BStone JBattestilli LRebelsky SShoop L(2024)Confidence vs Insight: Big and Rich Data in Computing Education ResearchProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630813(158-164)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630813

Index Terms

Pausing while programming: insights from keystroke analysis
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes
SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education

We study programming process data from two introductory programming courses. Between the course contexts, the programming languages differ, the teaching approaches differ, and the spoken languages differ. In both courses, students' keystroke data -- ...
A Method for Reducing Time-to-Solution in Quantum Annealing Through Pausing
HPCAsia '22: International Conference on High Performance Computing in Asia-Pacific Region

Recent research has shown that alternative annealing schedules provide the means for improving performance in modern quantum annealing devices. One such type of schedule is forward annealing with a pause, in which there is a period of time when system ...
A Framework that Explores the Cognitive Load of CS1 Assignments Using Pausing Behavior
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1

Pausing behavior in introductory Computer Science (CS1) courses has been related to course outcomes and could be linked to a student's cognitive load. Using Cognitive Load Theory and Vygotsky's Zone of Proximal Development as a theoretical framework, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEET '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training

May 2022

292 pages

ISBN:9781450392259

DOI:10.1145/3510456

Conference Chairs:
Cécile Péraire
Carnegie Mellon University
,
Alexander Serebrenik
Eindhoven University of Technology

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
293
Total Downloads

Downloads (Last 12 months)163
Downloads (Last 6 weeks)20

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Karakaš AHelic DMonga MLonati VBarendsen ESheard JPaterson J(2024)Combining Local Testing with Automatic Commits: Benefits for Progress Tracking and CS2 Students' Learning ExperienceProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653561(108-114)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653561
Brown NGuzdial MStephenson BStone JBattestilli LRebelsky SShoop L(2024)Confidence vs Insight: Big and Rich Data in Computing Education ResearchProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630813(158-164)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630813

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents