research-article

If They Build It, Will They Understand It? Exploring the Relationship between Student Code and Performance

Authors:

Diana FranklinAuthors Info & Claims

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

Pages 473 - 479

https://doi.org/10.1145/3341525.3387379

Published: 15 June 2020 Publication History

Abstract

The computer science community has struggled to assess student learning via Scratch programming at the primary school level (ages 7-12). Prior work has relied most heavily on artifact (student code/projects) analysis, with some attempts at one-on-one interviews and written assessments. In this paper, we explore the relationship between artifact analysis and written assessments. Through this study of a large-scale introductory computing implementation, we found that for students who had code in their projects, student performance on specific questions on the written assessments is only very weakly correlated to specific attributes of final projects typically used in artifact analysis as well as attributes we use to define candidate code (r < 0.2, p < 0.05). In particular, the correlation is not nearly strong enough to serve as a proxy for understanding.

References

[1]

Joel C Adams and Andrew R Webster. "What do students learn about programming from game, music video, and storytelling projects?" In: Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM. 2012,pp. 643--648.

[2]

Min-W. Wiebe E. Mott B. Boyer K. E. Lester J. Akram B. "Assessing Middle School Students' Computational Thinking Through Programming Trajectory Analysis". In: Proceedings of the 50th ACM technical symposium on Computer science education. ACM. 2019, p. 1269.

[3]

Michael T Battista. "Conceptualizations and issues related to learning progressions, learning trajectories, and levels of sophistication". In: The Mathematics Enthusiast 8.3 (2011), pp. 507--570.

[4]

Bryce Boe et al. "Hairball: Lint-inspired Static Analysis of Scratch Projects". In: Proceeding of the 44th ACM Technical Symposium on Computer Science Education. SIGCSE '13. Denver, Colorado, USA: ACM, 2013, pp. 215--220. isbn: 978--1--4503--1868--6. url: http://doi.acm.org/10.1145/2445196.2445265.

[5]

Karen Brennan and Mitchel Resnick. "New frameworks for studying and assessing the development of computational thinking". In: Proceedings of the 2012 annual meeting of the American Educational Research Association, Vancouver,Canada. Vol. 1. 2012, p. 25.

[6]

Creative Computing. An introductory computing curriculum using Scratch.

[7]

CS for ALL. url: https://www.csforall.org/.

[8]

Louise P Flannery et al. "Designing Scratch Jr: support for early childhood learning through computer programming". In: Proceedings of the 12th International Conference on Interaction Design and Children. ACM. 2013, pp. 1--10.

[9]

Diana Franklin et al. "Assessment of computer science learning in a scratch-based outreach program". In:Proceeding of the 44th ACM technical symposium on Computer science education. ACM. 2013, pp. 371--376.

[10]

Diana Franklin et al. "Using upper-elementary student performance to under-stand conceptual sequencing in a blocks-based curriculum". In: Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM. 2017, pp. 231--236.

[11]

David Hammer and TIFFANY-ROSE SIKORSKI. "Implications of complexity for research on learning progressions". In:Science Education99.3 (2015), pp. 424--431.

[12]

Dennis E Hinkle, William Wiersma, Stephen G Jurs, et al. "Applied statistics for the behavioral sciences". In: (1988).

[13]

Peter Hubwieser et al. "A global snapshot of computer science education in K-12 schools". In: Proceedings of the 2015 ITiCSE on working group reports. ACM. 2015, pp. 65--83.

[14]

Yifat Ben-David Kolikant. "Gardeners and cinema tickets: High school students' preconceptions of concurrency". In: Computer Science Education 11.3 (2001),pp. 221--245.

[15]

Gary Lewandowski et al. "Commonsense computing (episode 3): concurrency and concert tickets". In: Proceedings of the third international workshop on Computing education research. ACM. 2007, pp. 133--144.

[16]

Colleen M Lewis and Niral Shah. "Building upon and enriching grade four mathematics standards with programming curriculum". In:Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM. 2012, pp. 57--62.

Digital Library

[17]

Eva Marinus et al. "Unravelling the Cognition of Coding in 3-to-6-year Olds: The development of an assessment tool and the relation between coding abilityand cognitive compiling of syntax in natural language". In: Proceedings of the 2018 ACM Conference on International Computing Education Research. ACM. 2018, pp. 133--141.

[18]

Orni Meerbaum-Salant, Michal Armoni, and Mordechai Ben-Ari. "Learning computer science concepts with scratch". In: Computer Science Education 23.3(2013), pp. 239--264.

[19]

Robert J Mislevy and Geneva D Haertel. "Implications of evidence-centered design for educational testing". In: Educational Measurement: Issues and Practice 25.4 (2006), pp. 6--20.

[20]

Jesús Moreno-León et al. "On the Automatic Assessment of Computational Thinking Skills: A Comparison with Human Experts". In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. CHI EA '17. Denver, Colorado, USA: ACM, 2017, pp. 2788--2795.isbn: 978--1--4503--4656--6. url: http://doi.acm.org/10.1145/3027063.3053216.

[21]

Seymour Papert. Mindstorms: Children, computers, and powerful ideas. BasicBooks, Inc., 1980.

[22]

Alexander Repenning and Andri Ioannidou. "Broadening participation throughscalable game design". In: ACM SIGCSE Bulletin. Vol. 40. 1. ACM. 2008, pp. 305--309.

[23]

Kathryn M Rich et al. "K-8 learning trajectories derived from research literature:Sequence, repetition, conditionals". In: Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM. 2017, pp. 182--190.

[24]

Carsten Schulte. "Block Model: an educational model of program comprehension as a tool for a scholarly approach to teaching". In: Proceedings of the Fourth international Workshop on Computing Education Research. ACM. 2008, pp. 149--160.

Digital Library

[25]

Juha Sorva. "Notional Machines and Introductory Programming Education". In: ACM Transactions on Computing Education 13 (June 2013), 8:1--8:31.

Digital Library

[26]

Allison Elliott Tew and Mark Guzdial. "Developing a validated assessment of fundamental CS1 concepts". In: Proceedings of the 41st ACM technical symposium on Computer science education. ACM. 2010, pp. 97--101.

Digital Library

[27]

Ursula Wolz, Christopher Hallberg, and Brett Taylor. "Scrape: A tool for visualizing the code of Scratch programs". In: Poster presented at the 42nd ACM Technical Symposium on Computer Science Education, Dallas, TX. 2011.

Cited By

Lehtinen TKoutcheme CHellas ARoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension QuestionsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640058(221-232)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640058
Tran MKillen HPalmer JWeintrop DFranklin DStephenson BStone JBattestilli LRebelsky SShoop L(2024)Harmonizing Scratch Encore: Scaffolding K-8 Teachers in Customizing Culturally Responsive Computing MaterialsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630756(1335-1341)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630756
Woo KFalloon G(2023)The Search for Computer Science Concepts in Coding Animated Narratives: Tensions and OpportunitiesJournal of Educational Computing Research10.1177/0735633123117492961:7(1335-1358)Online publication date: 17-May-2023
https://doi.org/10.1177/07356331231174929
Show More Cited By

Index Terms

If They Build It, Will They Understand It? Exploring the Relationship between Student Code and Performance
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Personalized Assessment Worksheets for Scratch (PAWS): Exploring a Bridge between Interviews, Written Assessments, and Artifact Analysis
ICER '19: Proceedings of the 2019 ACM Conference on International Computing Education Research

The computer science community has struggled to assess student learning, especially at the early elementary level. Prior work has included one-on-one interviews, written assessments, and artifact analysis, each with their own benefits and drawbacks. ...
Assessment of computer science learning in a scratch-based outreach program
SIGCSE '13: Proceeding of the 44th ACM technical symposium on Computer science education

Many institutions have created and deployed outreach programs for middle school students with the goal of increasing the number and diversity of students who later pursue careers in computer science. While these programs have been shown to increase ...
Evaluation and Assessment Needs of Computing Education in Primary Grades
ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

Until recently, computer science (CS) has been predominantly taught at upper-secondary or tertiary levels. Lately, however, CS curricula have been introduced into school systems from the very first year of school. In this paper, we undertake a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

June 2020

615 pages

ISBN:9781450368742

DOI:10.1145/3341525

General Chairs:
Michail Giannakos
NTNU, Norway
,
Guttorm Sindre
Excited/NTNU, Norway
,
Program Chairs:
Andrew Luxton-Reilly
University of Auckland, NZ
,
Monica Divitini
NTNU, Norway

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ITiCSE '20

Sponsor:

SIGCSE

ITiCSE '20: Innovation and Technology in Computer Science Education

June 15 - 19, 2020

Trondheim, Norway

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
879
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)3

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lehtinen TKoutcheme CHellas ARoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension QuestionsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640058(221-232)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640058
Tran MKillen HPalmer JWeintrop DFranklin DStephenson BStone JBattestilli LRebelsky SShoop L(2024)Harmonizing Scratch Encore: Scaffolding K-8 Teachers in Customizing Culturally Responsive Computing MaterialsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630756(1335-1341)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630756
Woo KFalloon G(2023)The Search for Computer Science Concepts in Coding Animated Narratives: Tensions and OpportunitiesJournal of Educational Computing Research10.1177/0735633123117492961:7(1335-1358)Online publication date: 17-May-2023
https://doi.org/10.1177/07356331231174929
Lehtinen THaaranen LLeinonen J(2023)Automated Questionnaires About Students’ JavaScript Programs: Towards Gauging Novice Programming ProcessesProceedings of the 25th Australasian Computing Education Conference10.1145/3576123.3576129(49-58)Online publication date: 30-Jan-2023
https://dl.acm.org/doi/10.1145/3576123.3576129
Israel MLi JYan WElagha NHuggins-Manley CLuo FFranklin D(2023)How are Elementary Students Demonstrating Understanding of Decomposition within Elementary Mathematics?Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600143(222-235)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600143
Salac JEatinger DFranklin DDoyle MStephenson BDorn BSoh LBattestilli L(2023)The Role of Spatial Orientation in Diagram Design for Computational Thinking Development in K-8 TeachersProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569737(917-923)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569737
Saqr MLópez-Pernas SApiola M(2023)Capturing the Impact and the Chatter Around Computing Education Research Beyond Academia in Social Media, Patents, and BlogsPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_9(171-191)Online publication date: 18-Apr-2023
https://doi.org/10.1007/978-3-031-25336-2_9
Bastian MMühling A(2022)Comparing estimates of difficulty of programming constructsProceedings of the 22nd Koli Calling International Conference on Computing Education Research10.1145/3564721.3565950(1-12)Online publication date: 17-Nov-2022
https://dl.acm.org/doi/10.1145/3564721.3565950
Gonzalez-Maldonado DPugnali ATsan JEatinger DFranklin DWeintrop D(2022)Investigating the Use of Planning Sheets in Young Learners’ Open-Ended Scratch ProjectsProceedings of the 2022 ACM Conference on International Computing Education Research - Volume 110.1145/3501385.3543972(247-263)Online publication date: 3-Aug-2022
https://dl.acm.org/doi/10.1145/3501385.3543972
Cheng RDasgupta SHill B(2022)How Interest-Driven Content Creation Shapes Opportunities for Informal Learning in Scratch: A Case Study on Novices’ Use of Data StructuresProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502124(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502124
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents