research-article

Atoms of Confusion: The Eyes Do Not Lie

Authors:

Benedito de Oliveira,

Márcio Ribeiro,

José Aldo Silva da Costa,

Guilherme Amaral,

Rafael de Mello,

Anderson Oliveira,

Alessandro Garcia,

Rodrigo Bonifácio,

Baldoino FonsecaAuthors Info & Claims

SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering

Pages 243 - 252

https://doi.org/10.1145/3422392.3422437

Published: 21 December 2020 Publication History

Abstract

Code comprehension is crucial in software maintenance activities, though it can be hindered by misunderstandings and confusion patterns, namely, atoms of confusion. They are small pieces of code using specific programming language constructs, such as Conditional Operators and Comma Operators. A previous study showed that these atoms of confusion impact developers' performance, i.e., time and accuracy, and increase code misunderstandings. However, empirical knowledge of the impact of such atoms on code comprehension is still scarce, especially when it comes to analyzing that impact on developers' visual attention. The present study evaluates whether developers misunderstand the code in the presence of atoms of confusion with an eye tracker. For this purpose, we measure the tasks' completion time, the accuracy of the answers, and analyze the distribution of visual attention. We conducted a controlled experiment with 30 students and software practitioners. We ask the subjects to specify the output of three tasks with atoms and three without atoms randomly assigned using a Latin Square design. We use an eye-tracking camera to detect the visual attention of the participants while solving the tasks. From an aggregated perspective, we observed an increase of 43.02% in time and 36.8% in gaze transitions in code snippets with atoms. For accuracy, no statistically significant difference was observed. We also confirm that the regions that receive most of the eye attention were the regions with atoms. Our findings reinforce that atoms hinder developers' performance and comprehension. So, developers should avoid writing code with them.

References

[1]

Roman Bednarik and Markku Tukiainen. 2006. An Eye-tracking Methodology for Characterizing Program Comprehension Processes. In Proceedings of the Symposium on Eye Tracking Research & Applications (ETRA'06). 125--132.

Digital Library

[2]

George E. P. Box, J. Stuart Hunter, and William G. Hunter. 2005. Statistics for Experimenters: Design, innovation, and discovery. Wiley-Interscience.

[3]

Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In Proceedings of the International Conference on Program Comprehension (ICPC'15). 255--265.

Digital Library

[4]

Dan Gopstein, Jake Iannacone, Yu Yan, Lois Anne Delong, Yanyan Zhuang, Martin K.-C. Yeh, and Justin Cappos. 2017. Understanding Misunderstandings in Source Code. In Proceedings of the Foundations of Software Engineering (FSE'17). 129--139.

Digital Library

[5]

Dan Gopstein, Hongwei Henry Zhou, Phyllis Frankl, and Justin Cappos. 2018. Prevalence of Confusing Code in Software Projects: Atoms of Confusion in the Wild. In Proceedings of the Mining Software Repositories (MSR'18). 281--291.

Digital Library

[6]

Peter Gordon, Randall Hendrick, Marcus Johnson, and Yoonhyoung Lee. 2006. Similarity-based interference during language comprehension: Evidence from eye tracking during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 1304--1321.

[7]

Ahmad Jbara and Dror G Feitelson. 2017. How programmers read regular code: a controlled experiment using eye tracking. Empirical Software Engineering (EMSE'17) 22, 3 (2017), 1440--1477.

[8]

Marcel Just and Patricia Carpenter. 1965. A theory of reading: From eye fixations to comprehension. Psychological Review. Psychological Review, 329--354.

[9]

Romero Malaquias, Márcio Ribeiro, Rodrigo Bonifácio, Eduardo Monteiro, Flávio Medeiros, Alessandro Garcia, and Rohit Gheyi. 2017. The Discipline of Preprocessor-based Annotations Does #ifdef TAG N'T #endif Matter. In Proceedings of the International Conference on Program Comprehension (ICPC'17). 297--307.

Digital Library

[10]

Flávio Medeiros, Gabriel Lima, Guilherme Amaral, Sven Apel, Christian Kästner, Márcio Ribeiro, and Rohit Gheyi. 2019. An investigation of misunderstanding code patterns in C open-source software projects. Empirical Software Engineering (EMSE'19) 24, 4 (2019), 1693--1726.

[11]

Jean Melo, Claus Brabrand, and Andrzej Wkasowski. 2016. How Does the Degree of Variability Affect Bug Finding?. In Proceedings of the International Conference on Software Engineering (ICSE '16). 679--690.

Digital Library

[12]

Jean Melo, Fabricio Narcizo, Dan Hansen, Claus Brabrand, and Andrzej Wasowski. 2017. Variability through the Eyes of the Programmer. In Proceedings of the International Conference on Program Comprehension (ICPC '17). 34--44.

Digital Library

[13]

Roberto Minelli, Andrea Mocci, and Michele Lanza. 2015. I know what you did last summer-an investigation of how developers spend their time. In Proceedings of the International Conference on Program Comprehension (ICPC '15). 25--35.

Digital Library

[14]

Douglas C. Montgomery. 2006. Design and Analysis of Experiments. John Wiley & Sons.

Digital Library

[15]

Márcio Ribeiro, Paulo Borba, and Christian Kästner. 2014. Feature Maintenance with Emergent Interfaces. In Proceedings of the International Conference on Software Engineering (ICSE'14). 989--1000.

Digital Library

[16]

Carsten Schulte, Tony Clear, Ahmad Taherkhani, Teresa Busjahn, and James H. Paterson. 2010. An Introduction to Program Comprehension for Computer Science Educators. In Proceedings of the ITiCSE Working Group Reports. 65--86.

[17]

Zohreh Sharafi, Timothy Shaffer, Bonita Sharif, and Yann-Gaël Guéhéneuc. 2015. Eye-tracking metrics in software engineering. In Proceedings of the Asia-Pacific Software Engineering Conference (APSEC '15). 96--103.

[18]

Janet Siegmund. 2016. Program comprehension: Past, present, and future. In Proceedings of the Software Analysis, Evolution, and Reengineering (SANER'16), Vol. 5. 13--20.

[19]

Dag Sjoberg, Bente Anda, Erik Arisholm, Tore Dyba, Magne Jorgensen, Amela Karahasanovic, Espen Frimann Koren, and Marek Vokác. 2002. Conducting realistic experiments in software engineering. In Proceedings of the International Symposium on Empirical Software Engineering (ESEM'02). 17--26.

[20]

Leigh Ann Sudol-DeLyser, Mark Stehlik, and Sharon Carver. 2012. Code Comprehension Problems As Learning Events. In Proceedings of the Innovation and Technology in Computer Science Education (ITiCSE'12). 81--86.

Digital Library

[21]

Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.

[22]

Martin Yeh, Yu Yan, Dan Gopstein, and Yanyan Zhuang. 2017. Detecting and Comparing Brain Activity in Short Program Comprehension Using EEG. In Frontiers in Education Conference (FIE '17). 1--5.

Cited By

Shi GKazemi FGodfrey MMcIntosh S(2024)Reevaluating the Defect Proneness of Atoms of Confusion in Java SystemsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686677(154-164)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686677
Zid CZampetti FAntoniol GDi Penta MRoychoudhury APaiva AAbreu RStorey M(2024)A Study on the Pythonic Functional Constructs' UnderstandabilityProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639211(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639211
Grabinger LHauser FWolff CMottok J(2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03045-3
Show More Cited By

Index Terms

Atoms of Confusion: The Eyes Do Not Lie
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software
    2. Software verification and validation
  2. Software organization and properties
    1. Software system structures
      1. Abstraction, modeling and modularity

Recommendations

Thinking aloud about confusing code: a qualitative investigation of program comprehension and atoms of confusion
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Atoms of confusion are small patterns of code that have been empirically validated to be difficult to hand-evaluate by programmers. Previous research focused on defining and quantifying this phenomenon, but not on explaining or critiquing it. In this ...
Seeing confusion through a new lens: on the impact of atoms of confusion on novices’ code comprehension
Abstract
Code comprehension is crucial for software maintenance and evolution, but it can be hindered by tiny code snippets that can confuse the developers, called atoms of confusion. Previous studies investigated how atoms impact code comprehension ...
Eye tracking and program comprehension
EMIP '19: Proceedings of the 6th International Workshop on Eye Movements in Programming

Reading and comprehending code is different from reading regular text. And eye tracking, which measures where we look, for how long, and how much mental effort we exert, can provide crucial data. The talk will survey some achievements and suggest future ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering

October 2020

901 pages

ISBN:9781450387538

DOI:10.1145/3422392

General Chairs:
Everton Cavalcante
UFRN, Brazil
,
Francisco Dantas
UERN, Brazil
,
Thais Batista
UFRN, Brazil

Copyright © 2020 ACM.

© 2020 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

In-Cooperation

SBC: Brazilian Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SBES '20

SBES '20: 34th Brazilian Symposium on Software Engineering

October 21 - 23, 2020

Natal, Brazil

Acceptance Rates

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
155
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shi GKazemi FGodfrey MMcIntosh S(2024)Reevaluating the Defect Proneness of Atoms of Confusion in Java SystemsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686677(154-164)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686677
Zid CZampetti FAntoniol GDi Penta MRoychoudhury APaiva AAbreu RStorey M(2024)A Study on the Pythonic Functional Constructs' UnderstandabilityProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639211(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639211
Grabinger LHauser FWolff CMottok J(2024)On Eye Tracking in Software EngineeringSN Computer Science10.1007/s42979-024-03045-35:6Online publication date: 26-Jul-2024
https://dl.acm.org/doi/10.1007/s42979-024-03045-3
Zampetti FZid CAntoniol GPenta M(2024)The downside of functional constructs: a quantitative and qualitative analysis of their fix-inducing effectsEmpirical Software Engineering10.1007/s10664-024-10568-z30:1Online publication date: 22-Oct-2024
https://doi.org/10.1007/s10664-024-10568-z
Kuang PSöderberg ENiehorster DHöst MBavota GHao D(2023)Toward Gaze-Assisted Developer ToolsProceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER58687.2023.00015(49-54)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-NIER58687.2023.00015
Torres AOliveira COkimoto MMarcílio DQueiroga PCastor FBonifácio RCanedo ERibeiro MMonteiro E(2023)An Investigation of confusing code patterns in JavaScriptJournal of Systems and Software10.1016/j.jss.2023.111731203:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111731
da Costa JGheyi RCastor Fde Oliveira PRibeiro MFonseca B(2023)Seeing confusion through a new lens: on the impact of atoms of confusion on novices’ code comprehensionEmpirical Software Engineering10.1007/s10664-023-10311-028:4Online publication date: 18-May-2023
https://dl.acm.org/doi/10.1007/s10664-023-10311-0
Bogachenkova VNguyen LEbert FSerebrenik ACastor F(2022)Evaluating Atoms of Confusion in the Context of Code Reviews2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME55016.2022.00048(404-408)Online publication date: Oct-2022
https://doi.org/10.1109/ICSME55016.2022.00048

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents