Article

A methodology for analyzing the temporal evolution of novice programs based on semantic components

Authors:

Christopher D. Hundhausen,

Jonathan L. Brown,

Daniel SkarpasAuthors Info & Claims

ICER '06: Proceedings of the second international workshop on Computing education research

Pages 59 - 71

https://doi.org/10.1145/1151588.1151599

Published: 09 September 2006 Publication History

Abstract

Empirical studies of novice programming typically rely on code solutions or test responses as the basis of their analyses. While such data can provide insight into novice programming knowledge, they say little about the programming processes in which novices engage. For those interested in improving novice programming environments, a key research question arises: How can we collect and analyze data on novice programming that will enable us (a) to analyze and compare the programming processes promoted by alternative novice programming environments, and (b) ultimately to build better novice programming environments? To address this question, we have collected a large video corpus of novices as they construct code solutions in various versions of ALVIS Live! [17], a novice programming environment. Through detailed post-hoc analyses of our video corpus, we have developed a methodology for compiling the moment-by-moment evolution of novice code solutions. Based on an analysis of a model code solution's key semantic components, our methodology enables researchers to document, on a second-by-second basis, (a) what part of a code solution a programmer is focusing on, and (b) where the semantic feedback provided by the programming environment is helping. Although it is time and labor intensive, our methodology provides researchers with a standard set of data and representations for comparing the programming processes promoted by alternative programming environments.

References

[1]

Analyse-ItSoftwareLtd. Analyse-It® Statistical Add-In for Excel®, 2006, http://www.analyse-it.com.

[2]

Bakeman, R. and Gottman, J.M. Observing Iteraction: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge, England, 1996.

[3]

Brade, K., Guzdial, M., Steckel, M. and Soloway, E. Whorf: A visualization tool for software maintenance. In Proc. 1992 IEEE Workshop in Visual Languages, IEEE Computer Society Press, Los Alamitos, 1992, 148--154.

[4]

Brooks, R.E. Studying programmer behavior experimentally: the problems of proper methodology. Communications of the ACM, 23, 4, 1980, 208--213.

Digital Library

[5]

Brusilovsky, P., Calabrese, E., Hvorecky, J., Kouchnirenko, A. and Miller, P. Mini-languages: A way to learn programming principles. Education and Information Technologies, 2, 1997, 65--83.

Digital Library

[6]

Douglas, S.A., Hundhausen, C.D. and McKeown, D. Toward empirically-based software visualization languages. In Proceedings of the 11th IEEE Symposium on Visual Languages, IEEE Computer Society Press, Los Alamitos, CA, 1995, 342--349.

Digital Library

[7]

Dumas, J.S. and Redish, J.C. A Practical Guide to Usability Testing. Intellect Books, Portland, OR, 1993.

Digital Library

[8]

Ericsson, K.A. and Simon, H.A. Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA, 1984.

[9]

Ericsson, K.A. and Simon, H.A. Verbal reports as data. Psychological Review, 87, 1980, 215--251.

[10]

Fisher, C. Advancing the study of programming with computer-aided protocol analysis. In Empirical studies of programmers: Second Workshop, Ablex, Norwood, NJ, 1987, 198--216.

Digital Library

[11]

Gilmore, D.J. Methodological issues in the study of programming. In Hoc, J.-M., Green, T.R.G., Samurcay, R. and Gilmore, D.J. (eds.), Psychology of Programming, Academic Press, San Diego, 1990, 83--98.

[12]

Goldenson, D.R. and Wang, B.J. Use of Structure Editing Tools by Novice Programmers. In Empirical Studies of Programmers: Fourth Workshop, Ablex, 1991, 99--120.

[13]

Gray, W.D. and Anderson, J.R. Change-Episodes in Coding: When and How Do Programmers Change Their Code? In Empirical Studies of Programmers: Second Workshop, 1987, 185--197.

Digital Library

[14]

Green, T.R.G. and Petre, M. Usability analysis of visual programming environments: A 'cognitive dimensions' framework. Journal of Visual Langauges and Computing, 7, 2, 1996, 131--174.

[15]

Guzdial, M. Deriving software usage patterns from log files. Technical Report No. GIT-GVU-93-41, Graphics, Visualization, and Usability Center, Georgia Institute of Technology, 1993.

[16]

Hoc, J.-M. and Nguyen-Xuan, A. Language semantics, mental models and analogy. In Hoc, J.-M., Green, T.R.G., Samurcay, R. and Gilmore, D.J. (eds.), Psychology of Programming, Academic Press, San Diego, 1990, 139--156.

[17]

Hundhausen, C.D. and Brown, J.L. What You See Is What You Code: A 'Live' Algorithm Development and Visualization Environment for Novice Learners. Journal of Visual Languages and Computing, In press.

Digital Library

[18]

Kuittinen, M. and Sajaniemi, J. Teaching roles of variables in elementary programming courses. In Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education, ACM Press, New York, 2004, 57--61.

Digital Library

[19]

Norman, D.A. The Design of Everyday Things. Doubleday, New York, 1990.

Digital Library

[20]

OregonSocialLearningCenter. Coder, 2006, http://www.oslc.org/Training/download_coder.html.

[21]

Ormerod, T.C. and Ball, L.J. Does Programming Knowledge or Design Strategy Determine Shifts of Focus in Prolog Programming? In Empirical Studies of Programmers: Fifth Workshop, 1993, 162--186.

[22]

Romney, A.K., Weller, S.C. and Batchelder, W.H. Culture as consensus: A theory of culture and informant accuracy. American Anthropologist, 88, 2, 1986, 313--338.

[23]

Sanderson, P.M. and Fisher, C. Exploratory sequential data analysis: Foundations. Human-Computer Interaction, 9, 3-4, 1994, 251--318.

[24]

Sheil, B.A. The psychological study of programming. ACM Computing Surveys, 13, 1, 1981, 101--120.

Digital Library

[25]

Shneiderman, B. Direct manipulation: a step beyond programming languages. IEEE Computer, 16, 8, 1983, 57--69.

Digital Library

[26]

Shneiderman, B. Empirical Studies of Programmers: The Territory, Paths, and Destinations. In Empirical Studies of Programmers, 1986, 1--12.

Digital Library

[27]

Shrout, P.E. and Fleiss, J.L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 36, 2, 1979, 420--428.

[28]

Soloway, E. and Ehrlich, K. Empirical studies of programming knowledge. IEEE Transactions on Software Engineering, SE-10, 5, 1984, 595--609.

Digital Library

[29]

Spohrer, J.C. MARCEL: Simulating the novice programmer. Ablex, Norwood, NJ, 1992.

Digital Library

[30]

Spohrer, J.C. and Soloway, E. Analyzing the High Frequency Bugs in Novice Programs. In Empirical Studies of Programmers, 1986, 230--251.

Digital Library

[31]

TechSmith. An overview of Morae: A breakthrough in usability testing and user experience research, 2006, http://www.techsmith.com/morae/whitepaper/overview.asp.

Cited By

Kennedy CKraemer EScharlau BMcDermott RPears ASabin M(2019)Qualitative Observations of Student ReasoningProceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education10.1145/3304221.3319751(224-230)Online publication date: 2-Jul-2019
https://dl.acm.org/doi/10.1145/3304221.3319751
Cook MFowler MHallstrom JHollingsworth JSchwab TSun YSitaraman MPolycarpou IRead JAndreou PArmoni M(2018)Where exactly are the difficulties in reasoning logically about code? experimentation with an online systemProceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education10.1145/3197091.3197133(39-44)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3197091.3197133
Bao LXing ZXia XLo DHassan A(2018)Inference of development activities from interaction with uninstrumented applicationsEmpirical Software Engineering10.1007/s10664-017-9547-823:3(1313-1351)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10664-017-9547-8
Show More Cited By

Index Terms

A methodology for analyzing the temporal evolution of novice programs based on semantic components
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
        Information science education

Recommendations

An experimental study of the impact of visual semantic feedback on novice programming

Prior empirical studies of programming have shown that novice programmers tend to program by exploration, relying on frequent compilation and execution of their code in order to make progress. One way visual and end-user programming environments have ...
A novice's process of object-oriented programming
OOPSLA '06: Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications

Exposing students to the process of programming is merely implied but not explicitly addressed in texts on programming which appear to deal with 'program' as a noun rather than as a verb.We present a set of principles and techniques as well as an ...
Evaluating assessments of novice programming environments
ICER '05: Proceedings of the first international workshop on Computing education research

Educators have developed a myriad of tools to help novices learn to program. Unfortunately, too little is known about the educational impact of these environments, or even how to assess this impact. In this paper we summarize a representative collection ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '06: Proceedings of the second international workshop on Computing education research

September 2006

144 pages

ISBN:1595934944

DOI:10.1145/1151588

Program Chairs:
Richard Anderson
University of Washington, Seattle, USA
,
Sally A. Fincher
University of Kent at Canterbury, UK
,
Mark Guzdial
Georgia Institute of Technology, Atlanta, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICER06

Sponsor:

ICER06: International Computing Education Research Workshop 2006

September 9 - 10, 2006

Canterbury, United Kingdom

Acceptance Rates

Overall Acceptance Rate 189 of 803 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
397
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kennedy CKraemer EScharlau BMcDermott RPears ASabin M(2019)Qualitative Observations of Student ReasoningProceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education10.1145/3304221.3319751(224-230)Online publication date: 2-Jul-2019
https://dl.acm.org/doi/10.1145/3304221.3319751
Cook MFowler MHallstrom JHollingsworth JSchwab TSun YSitaraman MPolycarpou IRead JAndreou PArmoni M(2018)Where exactly are the difficulties in reasoning logically about code? experimentation with an online systemProceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education10.1145/3197091.3197133(39-44)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3197091.3197133
Bao LXing ZXia XLo DHassan A(2018)Inference of development activities from interaction with uninstrumented applicationsEmpirical Software Engineering10.1007/s10664-017-9547-823:3(1313-1351)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10664-017-9547-8
Carter AHundhausen CAdesope O(2017)Blending Measures of Programming and Social Behavior into Predictive Models of Student Achievement in Early Computing CoursesACM Transactions on Computing Education10.1145/312025917:3(1-20)Online publication date: 28-Aug-2017
https://dl.acm.org/doi/10.1145/3120259
Bao LLi JXing ZWang XXia XZhou B(2017)Extracting and analyzing time-series HCI data from screen-captured task videosEmpirical Software Engineering10.1007/s10664-015-9417-122:1(134-174)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s10664-015-9417-1
Carter AHundhausen CAdesope ODorn BSheard JCutts Q(2015)The Normalized Programming State ModelProceedings of the eleventh annual International Conference on International Computing Education Research10.1145/2787622.2787710(141-150)Online publication date: 9-Jul-2015
https://dl.acm.org/doi/10.1145/2787622.2787710
Lingfeng Bao Jing Li Xing ZXinyu Wang Bo Zhou (2015)Reverse engineering time-series interaction data from screen-captured videos2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)10.1109/SANER.2015.7081850(399-408)Online publication date: Mar-2015
https://doi.org/10.1109/SANER.2015.7081850
Rodrigo MAndallaza TCastro FArmenta MDy TJadud M(2014)An Analysis of Java Programming Behaviors, Affect, Perceptions, and Syntax Errors among Low-Achieving, Average, and High-Achieving Novice ProgrammersJournal of Educational Computing Research10.2190/EC.49.3.b49:3(293-325)Online publication date: 4-Apr-2014
https://doi.org/10.2190/EC.49.3.b
Shekhar VPrabhu APuranik KAntin LKumar V(2014)JFLAP Extensions for Instructors and StudentsProceedings of the 2014 IEEE Sixth International Conference on Technology for Education10.1109/T4E.2014.22(140-143)Online publication date: 18-Dec-2014
https://dl.acm.org/doi/10.1109/T4E.2014.22
Fuchs MHeckner MRaab FWolff C(2014)Monitoring students' mobile app coding behavior data analysis based on IDE and browser interaction logs2014 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON.2014.6826202(892-899)Online publication date: Apr-2014
https://doi.org/10.1109/EDUCON.2014.6826202
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents