review-article

Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer

Authors:

Raymond Lister,

Dennis J. Bouvier,

Robert McCartney,

Otto Seppälä,

Errol ThompsonAuthors Info & Claims

ACM SIGCSE Bulletin, Volume 41, Issue 4

Pages 156 - 173

https://doi.org/10.1145/1709424.1709460

Published: 18 January 2010 Publication History

Abstract

In New Zealand and Australia, the BRACElet project has been investigating students' acquisition of programming skills in introductory programming courses. The project has explored students' skills in basic syntax, tracing code, understanding code, and writing code, seeking to establish the relationships between these skills. This ITiCSE working group report presents the most recent step in the BRACElet project, which includes replication of earlier analysis using a far broader pool of naturally occurring data, refinement of the SOLO taxonomy in code-explaining questions, extension of the taxonomy to code-writing questions, extension of some earlier studies on students' 'doodling' while answering exam questions, and exploration of a further theoretical basis for work that until now has been primarily empirical.

References

[1]

Baroody, A., Feil, Y., and Johnson, A. (2007). An alternative reconceptualization of procedural and conceptual knowledge. Journal for Research in Mathematics Education, 38:2, 115--131.

[2]

Biggs, J. B. (1999). Teaching for quality learning at university. Buckingham: Open University Press.

[3]

Biggs, J. B. and Collis, K. F. (1982). Evaluating the quality of learning: the SOLO taxonomy (Structure of the Observed Learning Outcome). New York: Academic Press.

[4]

Byckling, P. and Sajaniemi, J. (2006). Roles of variables and programming skills improvement. SIGCSE Bulletin 38:1, 413--417.

Digital Library

[5]

Clear, T., Edwards, J., Lister, R., Simon, B., Thompson, E. and Whalley, J. (2008). The teaching of novice computer programmers: bringing the scholarly-research approach to Australia. Tenth Australasian Computing Education Conference (ACE 2008), Wollongong, Australia, 63--68.

Digital Library

[6]

Clear, T., Whalley, J., Lister, R., Carbone, A., Hu, M., Sheard, J., Simon, B. and Thompson, E. (2008). Reliably classifying novice programmer exam results using the SOLO taxonomy. 21st Annual NACCQ Conference, NACCQ, Auckland, New Zealand, 23--30.

[7]

Colburn, T. and Shute, G. (2007). Abstraction in computer science. Minds & Machines 17, 169--184.

Digital Library

[8]

Cottrill, J., Dubinsky, E., Nichols, D., Schwingendorf, K., Thomas, K., and Vidakovic, D. (1996). Understanding the limit concept: beginning with a coordinated process scheme. Journal of Mathematical Behavior 15:2, 167--192.

[9]

Denny, P., Luxton-Reilly, A. and Simon, B. (2008). Evaluating a new exam question: Parsons problems. Fourth International Workshop on Computing Education Research (ICER '08), Sydney, Australia, 113--124.

Digital Library

[10]

Fincher, S., Lister, R., Clear, T., Robins, A., Tenenberg, J. and Petre, M. (2005). Multi-institutional, multi-national studies in CSEd research: some design considerations and trade-offs. in Anderson, R., Fincher, S. and Guzdial, M. eds. First International Workshop on Computing Education Research, Seattle, WA, USA, 111--121.

Digital Library

[11]

Gray, E., Pinto, M., Pitta, D., and Tall, D. (1999). Knowledge construction and diverging thinking in elementary and advanced mathematics. Educational Studies in Mathematics 38, 111--133.

[12]

Gray, E. and Tall, D. (2007). Abstraction as a natural process of mental compression. Mathematics Education Research Journal 19:2, 23--40.

[13]

Hattie, J. and Purdie, N. (1998). The SOLO model: addressing fundamental measurement issues. In Dart, B. & Boulton-Lewis, G. M. (Eds.), Teaching and learning in higher education, 145--176. Camberwell, Vic: Australian Council of Educational Research.

[14]

Hiebert, J. and Lefevre, P. (1986). Conceptual and procedural knowledge in mathematics: an introductory analysis. In Hiebert, J., ed., Conceptual and procedural knowledge: the case of mathematics, 1--27. Erlbaum, Hillsdale, NJ.

[15]

Lister R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, E., Sanders, K., Seppälä, O., Simon, B., and Thomas, L. (2004). A multi national study of reading and tracing skills in novice programmers. SIGCSE Bulletin 36:4, 119--150.

Digital Library

[16]

Lister, R., Fidge, C., and Teague, D. (2009). Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. 14th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE'09), July 3-8, 2009, Paris, France.

Digital Library

[17]

Lister, R., Simon, B., Thompson, E., Whalley, J. L., and Prasad, C. (2006). Not seeing the forest for the trees: novice programmers and the SOLO taxonomy. SIGCSE Bulletin 38:3, 118--122.

Digital Library

[18]

Lopez, M., Whalley, J., Robbins, P., and Lister, R. (2008). Relationships between reading, tracing and writing skills in introductory programming. Fourth International Workshop on Computing Education Research (ICER '08), Sydney, Australia, 101--112.

Digital Library

[19]

McCartney, R., Mostrom, J.E., Sanders, K., and Seppälä, O. (2004). Questions, annotations, and institutions: observations from a study of novice programmers. Fourth Finnish / Baltic Sea Conference on Computer Science Education (Koli Calling 2004), 11--19.

[20]

McCormick, R. (1997). Conceptual and procedural knowledge. International Journal of Technology and Design Education 7:1-2, 141--159.

[21]

Parsons, D. and Haden, P. (2006). Parson's programming puzzles: a fun and effective learning tool for first programming courses. Eighth Australasian Computing Education Conference (ACE2006), Hobart, Australia, 157--163.

Digital Library

[22]

Pegg, J. (2002). Assessment in mathematics: A developmental approach. In J. M. Royer (Ed.), Mathematical Cognition 227--259. Information Age Publishing: Greenwich, CT, USA.

[23]

Pegg, J. and Tall, D. (2005). The fundamental cycle of concept construction underlying various theoretical frameworks. ZDM 37:6, 468--474.

[24]

Perkins, D. and Martin, F. (1989). Fragile knowledge and neglected strategies in novice programmers. In Soloway, E. and Spohrer, J, Eds., Studying the Novice Programmer. Lawrence Erlbaum Associates, Hillsdale, NJ, USA, 213--229.

Digital Library

[25]

Philpott, A., Robbins, P., and Whalley, J. (2007). Accessing the steps on the road to relational thinking. 20th Annual Conference of the National Advisory Committee on Computing Qualifications, Nelson, New Zealand, 286.

Digital Library

[26]

Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests (Copenhagen, Danish Institute for Educational Research), expanded edition (1980) with foreword and afterword by B.D. Wright. Chicago: The University of Chicago Press.

[27]

Robins, A., Rountree, J., and Rountree, N. (2003). Learning and teaching programming: a review and discussion. Computer Science Education 13:2, 137--172.

[28]

Sfard, A. (1988). Operational vs. structural method of teaching mathematics -- case study. Twelfth Conference for the Psychology of Mathematics Education 2, 560--567. Vesprèm, Hungary: Ferenc Genzwein, OOK.

[29]

Sfard, A. (1991). On the dual nature of mathematical conceptions: reflections on processes and objects as different sides of the same coin. Educational Studies in Mathematics 22, 1--36.

[30]

Sheard, J., Carbone, A., Lister, R., Simon, B., Thompson, E., and Whalley, J. L. (2008). Going SOLO to assess novice programmers. SIGCSE Bulletin 40:3, 209--213.

Digital Library

[31]

Simon (2009). Code-explaining exam questions: a cautionary note. Ninth International Conference on Computing Education Research (Koli Calling 2009).

[32]

Soloway, E. (1986). Learning to program = learning to construct mechanisms and explanations. Communications of the ACM 29:9, 850--858.

Digital Library

[33]

Star, J. R. (2005). Reconceptualizing procedural knowledge. Journal for Research in Mathematics Education 36, 401--411.

[34]

Tall, D., (2008). The transition to formal thinking in mathematics. Mathematics Education Research Journal 20:2, 5--24.

[35]

Thompson, E. (2004). Does the sum of the parts equal the whole? 21st Annual NACCQ Conference, NACCQ, Auckland, New Zealand, 2008, 440--445.

[36]

Thompson, E. (2007). Holistic assessment criteria -- Applying SOLO to programming projects. Ninth Australasian Computing Education Conference (ACE2007), Ballarat, Australia, 155--162.

Digital Library

[37]

Thompson, E. (2009). How do they understand? Practitioner perceptions of an object-oriented program. Dissertation, Massey University, Palmerston North.

[38]

Venables, A., Tan, G., and Lister, R. (2009) A closer look at tracing, explaining and code writing skills in the novice programmer. Fifth International Workshop on Computing Education Research (ICER '09), Berkeley, California, USA.

Digital Library

[39]

Wiedenbeck, S. (1985). Novice/expert differences in programming skills. International Journal of Man-Machine Studies 23:4, 383--390.

Digital Library

[40]

Whalley, J. L., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, P. K. A., and Prasad, C. (2006). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Eighth Australasian Computing Education Conference (ACE2006), Hobart, Australia, 243--252.

Digital Library

[41]

Whalley, J. and Lister, R. (2009). The BRACElet 2009.1 (Wellington) Specification. Eleventh Australasian Computing Education Conference (ACE2009), Wellington, New Zealand, 9--18.

Digital Library

[42]

Whalley, J. and Robbins, P. (2007). Report on the Fourth BRACElet Workshop. Bulletin of Applied Computing and IT. Retrieved June 7, 2007 from http://www.naccq.co.nz/bacit/0501/2007Whalley_BRACELET_Workshop.htm.

[43]

Wright, B.D, and Linacre, J.M. (1994) Reasonable meansquare fit values. Rasch Measurement Transactions 8:3, 370.

Cited By

Lehtinen TKoutcheme CHellas ARoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension QuestionsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640058(221-232)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640058
Haaranen LAhrenberg LHellas A(2023)Decades of Striving for Pedagogical and Technological AlignmentProceedings of the 23rd Koli Calling International Conference on Computing Education Research10.1145/3631802.3631809(1-8)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3631802.3631809
Smith DFowler MZilles CLaakso MMonga MSimon Sheard J(2023)Investigating the Role and Impact of Distractors on Parsons Problems in CS1 AssessmentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588819(417-423)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588819
Show More Cited By

Index Terms

Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer
1. Applied computing
  1. Education
2. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

A closer look at tracing, explaining and code writing skills in the novice programmer
ICER '09: Proceedings of the fifth international workshop on Computing education research workshop

The way in which novice programmers learn to write code is of considerable interest to computing education researchers. One research approach to understanding how beginners acquire their programming abilities has been to look at student performance in ...
Relationships between reading, tracing and writing skills in introductory programming
ICER '08: Proceedings of the Fourth international Workshop on Computing Education Research

This study analyzed student responses to an examination, after the students had completed one semester of instruction in programming. The performance of students on code tracing tasks correlated with their performance on code writing tasks. A ...
Further evidence of a relationship between explaining, tracing and writing skills in introductory programming
ITiCSE '09: Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education

This paper reports on a replication of earlier studies into a possible hierarchy of programming skills. In this study, the students from whom data was collected were at a university that had not provided data for earlier studies. Also, the students were ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGCSE Bulletin

ACM SIGCSE Bulletin Volume 41, Issue 4

December 2009

205 pages

ISSN:0097-8418

DOI:10.1145/1709424

Issue’s Table of Contents

Copyright © 2010 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 January 2010

Published in SIGCSE Volume 41, Issue 4

Check for updates

Author Tags

Qualifiers

Review-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

77
Total Citations
View Citations
937
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)7

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lehtinen TKoutcheme CHellas ARoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension QuestionsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640058(221-232)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640058
Haaranen LAhrenberg LHellas A(2023)Decades of Striving for Pedagogical and Technological AlignmentProceedings of the 23rd Koli Calling International Conference on Computing Education Research10.1145/3631802.3631809(1-8)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3631802.3631809
Smith DFowler MZilles CLaakso MMonga MSimon Sheard J(2023)Investigating the Role and Impact of Distractors on Parsons Problems in CS1 AssessmentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588819(417-423)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588819
Kohn TKomm DLaakso MMonga MSimon Sheard J(2023)Coping With Scoping: Understanding Scope and ParametersProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588798(201-207)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588798
Smith, IV DPoulsen SFowler MZilles CChoppella VPhatak DLuxton-Reilly ACraig M(2023)Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 AssessmentsProceedings of the ACM Conference on Global Computing Education Vol 110.1145/3576882.3617927(154-160)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3576882.3617927
Smith DZilles CDoyle MStephenson BDorn BSoh LBattestilli L(2023)Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569801(924-930)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569801
Sheng YLi BWu ZZhong PDuan G(2023)A C Language Learning Platform Based on Parsons ProblemsComputer Science and Education10.1007/978-981-99-2446-2_49(541-552)Online publication date: 14-May-2023
https://doi.org/10.1007/978-981-99-2446-2_49
Ericson BDenny PPrather JDuran RHellas ALeinonen JMiller CMorrison BPearce JRodger SQuille KAlshaigy BBecker BQuille KLaakso M(2022)Parsons Problems and BeyondProceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3571785.3574127(191-234)Online publication date: 27-Dec-2022
https://dl.acm.org/doi/10.1145/3571785.3574127
Stankov EJovanov MMadevska Bogdanova A(2022)Smart generation of code tracing questions for assessment in introductory programmingComputer Applications in Engineering Education10.1002/cae.2256731:1(5-25)Online publication date: 19-Sep-2022
https://doi.org/10.1002/cae.22567
Shi N(2021)Improving Undergraduate Novice Programmer Comprehension through Case-Based Teaching with Roles of Variables to Provide ScaffoldingInformation10.3390/info1210042412:10(424)Online publication date: 16-Oct-2021
https://doi.org/10.3390/info12100424
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents