research-article

Testing scratch programs automatically

Authors:

Andreas Stahlbauer,

Gordon FraserAuthors Info & Claims

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 165 - 175

https://doi.org/10.1145/3338906.3338910

Published: 12 August 2019 Publication History

Abstract

Block-based programming environments like Scratch foster engagement with computer programming and are used by millions of young learners. Scratch allows learners to quickly create entertaining programs and games, while eliminating syntactical program errors that could interfere with progress. However, functional programming errors may still lead to incorrect programs, and learners and their teachers need to identify and understand these errors. This is currently an entirely manual process. In this paper, we introduce a formal testing framework that describes the problem of Scratch testing in detail. We instantiate this formal framework with the Whisker tool, which provides automated and property-based testing functionality for Scratch programs. Empirical evaluation on real student and teacher programs demonstrates that Whisker can successfully test Scratch programs, and automatically achieves an average of 95.25 % code coverage. Although well-known testing problems such as test flakiness also exist in the scenario of Scratch testing, we show that automated and property-based testing can accurately reproduce and replace the manually and laboriously produced grading efforts of a teacher, and opens up new possibilities to support learners of programming in their struggles.

References

[1]

E. Aivaloglou and F. Hermans. 2016. How kids code and how we know: An exploratory study on the Scratch repository. In Proc. ICER. ACM, 53–61.

Digital Library

[2]

G. Anielak, G. Jakacki, and S. Lasota. 2015. Incremental test case generation using bounded model checking: an application to automatic rating. STTT 17, 3 (2015), 339–349.

Digital Library

[3]

H. Attiya, A. Bar-Noy, and D. Dolev. 1995. Sharing Memory Robustly in Message-Passing Systems. J. ACM 42, 1 (1995), 124–142.

Digital Library

[4]

D. Bau, J. Gray, C. Kelleher, J. Sheldon, and F. A. Turbak. 2017. Learnable programming: blocks and beyond. Commun. ACM 60, 6 (2017), 72–80.

Digital Library

[5]

J. Bengtsson and W. Yi. 2003. Timed Automata: Semantics, Algorithms and Tools. In Lectures on Concurrency and Petri Nets (LNCS 3098). Springer, 87–124.

[6]

D. Beyer, M. Dangl, D. Dietsch, M. Heizmann, and A. Stahlbauer. 2015. Witness validation and stepwise testification across software verifiers. In Proc. ESEC/FSE. ACM, 721–733.

Digital Library

[7]

B. Boe, C. Hill, M. Len, G. Dreschler, P. Conrad, and D. Franklin. 2013. Hairball: Lint-inspired Static Analysis of Scratch Projects. In Proc. SIGCSE (SIGCSE ’13). ACM, New York, NY, USA, 215–220.

Digital Library

[8]

K. Claessen and J. Hughes. 2011. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. SIGPLAN Not. 46, 4 (2011), 53–64.

Digital Library

[9]

C. Delporte-Gallet, H. Fauconnier, and R. Guerraoui. 2003. Shared memory vs message passing. Technical Report.

[10]

E. W. Dijkstra. 1970. Notes on Structured Programming.

[11]

C. Benac Earle, L.-A. Fredlund, and J. Hughes. 2016. Automatic Grading of Programming Exercises using Property-Based Testing. In Proc. ITiCSE. ACM, 47–52.

Digital Library

[12]

LEGO Education. {n.d.}. LEGO Education. https://education.lego.com.

[13]

M.Y. Feng and A. McAllister. 2006. A tool for automated GUI program grading. In Proc. FIE. IEEE, 7–12.

[14]

J. -C. Fernandez, L. Mounier, and C. Pachon. 2003. Property Oriented Test Case Generation. In Proc. FATES (LNCS 2931). Springer, 147–163.

[15]

G. E. Forsythe and N. Wirth. 1965. Automatic grading programs. Commun. ACM 8, 5 (1965), 275–278.

Digital Library

[16]

Micro:bit Educational Foundation. {n.d.}. Micro:bit. https://microbit.org/.

[17]

Raspberry Pi Foundation. {n.d.}. Code Club. https://codeclubprojects.org/.

[18]

P. Godefroid, A. Kiezun, and M. Y. Levin. 2008. Grammar-based whitebox fuzzing. In Proc. PLDI. ACM, 206–215.

Digital Library

[19]

F. Gross, G. Fraser, and A. Zeller. 2012. Search-based System Testing: High Coverage, No False Alarms. In Proc. ISSTA (ISSTA 2012). ACM, New York, NY, USA, 67–77.

Digital Library

[20]

T. A. Henzinger, R. Jhala, R. Majumdar, G. C. Necula, G. Sutre, and W. Weimer. 2002. Temporal-Safety Proofs for Systems Code. In Proc. CAV (LNCS 2404). Springer, 526–538.

Digital Library

[21]

F. Hermans and E. Aivaloglou. 2016. Do code smells hamper novice programming? A controlled experiment on Scratch programs. In Proc. ICPC. IEEE, 1–10.

[22]

P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Koli Calling. ACM, 86–93.

Digital Library

[23]

D. E. Johnson. 2016. ITCH: Individual Testing of Computer Homework for Scratch Assignments. In Proc. SIGCSE. ACM, New York, NY, USA, 223–227.

Digital Library

[24]

S. Katz. 2006. Aspect Categories and Classes of Temporal Properties. (2006), 106–134.

Digital Library

[25]

S. Konur. 2013. A survey on temporal logics for specifying and verifying real-time systems. Frontiers Comput. Sci. 7, 3 (2013), 370–403.

Digital Library

[26]

O. Kupferman, N. Piterman, and M. Y. Vardi. 2009. From liveness to promptness. Formal Methods in System Design 34, 2 (2009), 83–103.

Digital Library

[27]

S. K. Lahiri, R. Nieuwenhuis, and A. Oliveras. 2006. SMT Techniques for Fast Predicate Abstraction. In Proc. CAV (LNCS 4144). Springer, 424–437.

Digital Library

[28]

A. Lester, M.Schwern, and A. Armstrong. {n.d.}. The Test Anything Protocol v13. https://testanything.org/tap-version-13-specification.html.

[29]

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 643–653.

Digital Library

[30]

O. Maler, D. Nickovic, and A. Pnueli. 2007. On Synthesizing Controllers from Bounded-Response Properties. In Proc. CAV (LNCS 4590). Springer, 95–107.

Digital Library

[31]

J. Maloney, M. Resnick, N. Rusk, B. Silverman, and E. Eastmond. 2010. The Scratch Programming Language and Environment. TOCE 10, 4 (2010), 16:1–16:15.

Digital Library

[32]

T. Matlock, M. Ramscar, and L. Boroditsky. 2005. On the Experiential Link Between Spatial and Temporal Language. Cognitive Science 29, 4 (2005), 655–664.

[33]

A. W. Mazurkiewicz. 1986. Trace Theory. In Advances in Petri Nets (LNCS 255). Springer, 279–324.

Digital Library

[34]

A. M. Memon. 2007. An event-flow model of GUI-based applications for testing. Softw. Test., Verif. Reliab. 17, 3 (2007), 137–157.

[35]

N. Mirzaei, S. Malek, C. S. Păsăreanu, N. Esfahani, and R.Mahmood. 2012. Testing android apps through symbolic execution. ACM SIGSOFT Software Engineering Notes 37, 6 (2012), 1–5.

Digital Library

[36]

J. Moreno-León and G. Robles. 2015. Dr. Scratch: A Web Tool to Automatically Evaluate Scratch Projects (Proc. WiPSCE). ACM, New York, NY, USA, 132–133.

Digital Library

[37]

F. Palomba and A. Zaidman. 2017. Does Refactoring of Test Smells Induce Fixing Flaky Tests?. In Proc. ICSME. IEEE Computer Society, 1–12.

[38]

M. Pezzè, P. Rondena, and D. Zuddas. 2018. Automatic GUI testing of desktop applications: an empirical assessment of the state of the art. In ISSTA/ECOOP Workshops. ACM, 54–62.

Digital Library

[39]

A. Pnueli and A. Zaks. 2008. On the Merits of Temporal Testers. In 25 Years of Model Checking (LNCS 5000). Springer, 172–195.

Digital Library

[40]

Y. S. Ramakrishna, P. M. Melliar-Smith, L. E. Moser, L. K. Dillon, and G. Kutty. 1993. Really visual temporal reasoning. In RTSS. IEEE Computer Society, 262–273.

[41]

Gregorio Robles, Jesús Moreno-León, Efthimia Aivaloglou, and Felienne Hermans. 2017. Software clones in scratch projects: On the presence of copy-and-paste in computational thinking learning. In 2017 IEEE 11th International Workshop on Software Clones (IWSC). IEEE, 1–7.

[42]

A. W. Roscoe. 2001. Compiling shared variable programs into CSP. In Proc. PROGRESS, Vol. 2001.

[43]

Y. Sun and E. L. Jones. 2004. Specification-driven automated testing of GUI-based Java programs. In Proc. 42nd annual Southeast regional conference. ACM, 140–145.

Digital Library

[44]

M. Sung, S. Kim, S. Park, N. Chang, and H. Shin. 2002. Comparative performance evaluation of Java threads for embedded applications: Linux Thread vs. Green Thread. Inf. Process. Lett. 84, 4 (2002), 221–225.

Digital Library

[45]

Alaaeddin Swidan, Alexander Serebrenik, and Felienne Hermans. 2017. How do Scratch programmers name variables and procedures?. In 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 51–60.

Cited By

Steckenbiller SFeldmeier PFraser GHeuer UObermüller F(2024)Introducing Block-Based Testing in ScratchProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699589(1-2)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699589
Georg Barth MThorgeirsson SSu Z(2024)A Direct Manipulation Programming Environment for Teaching Introductory and Advanced Software TestingProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699564(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699564
Feldmeier PFraser GHeuer UObermüller FSteckenbiller S(2024)A Block-Based Testing Framework for ScratchProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699547(1-12)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699547
Show More Cited By

Index Terms

Testing scratch programs automatically
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

A Block-Based Testing Framework for Scratch
Koli Calling '24: Proceedings of the 24th Koli Calling International Conference on Computing Education Research
Block-based programming environments like Scratch are widely used in introductory programming courses. They facilitate learning pivotal programming concepts by eliminating syntactical errors, but logical errors that break the desired program behaviour are ...
Introducing Block-Based Testing in Scratch
Koli Calling '24: Proceedings of the 24th Koli Calling International Conference on Computing Education Research
The block-based programming environment Scratch is widely used by learners to make their first steps into the world of programming. Often applied in educational contexts, both teachers and students would benefit from quick and individual feedback on their ...
SnapCheck: Automated Testing for Snap! Programs
ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1

Programming environments such as Snap, Scratch, and Processing engage learners by allowing them to create programming artifacts such as apps and games, with visual and interactive output. Learning programming with such a media-focused context has been ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2019

1264 pages

ISBN:9781450355728

DOI:10.1145/3338906

General Chairs:
Marlon Dumas
University of Tartu, Estonia
,
Dietmar Pfahl
University of Tartu, Estonia
,
Program Chairs:
Sven Apel
Saarland University, Germany
,
Alessandra Russo
Imperial College, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

DFG
EPSRC

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 30, 2019

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
788
Total Downloads

Downloads (Last 12 months)114
Downloads (Last 6 weeks)11

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Steckenbiller SFeldmeier PFraser GHeuer UObermüller F(2024)Introducing Block-Based Testing in ScratchProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699589(1-2)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699589
Georg Barth MThorgeirsson SSu Z(2024)A Direct Manipulation Programming Environment for Teaching Introductory and Advanced Software TestingProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699564(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699564
Feldmeier PFraser GHeuer UObermüller FSteckenbiller S(2024)A Block-Based Testing Framework for ScratchProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699547(1-12)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699547
Obermüller FFraser G(2024)Do Scratchers Fix Their Bugs? Detecting Fixes of Scratch Static Analysis WarningsProceedings of the 19th WiPSCE Conference on Primary and Secondary Computing Education Research10.1145/3677619.3678108(1-4)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3677619.3678108
Sun JSu TSun JLi JWang MPu Gd'Amorim M(2024)Property-Based Testing for Validating User Privacy-Related Functionalities in Social Media AppsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663863(440-451)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663863
Nurue HGray JLo DGamess E(2024)A Testing Extension for ScratchProceedings of the 2024 ACM Southeast Conference10.1145/3603287.3651217(266-271)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3603287.3651217
Deiner AFraser GRoychoudhury APaiva AAbreu RStorey M(2024)NuzzleBug: Debugging Block-Based Programs in ScratchProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623331(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623331
Strijbol NDe Proft RGoethals KMesuere BDawyndt PScholliers C(2024)Blink: An educational software debugger for ScratchSoftwareX10.1016/j.softx.2023.10161725(101617)Online publication date: Feb-2024
https://doi.org/10.1016/j.softx.2023.101617
Greifenstein LBrune MFuchs THeuer UFraser G(2023)Impact of Hint Content on Performance and Learning: A Study with Primary School Children in a Scratch CourseProceedings of the 18th WiPSCE Conference on Primary and Secondary Computing Education Research10.1145/3605468.3605498(1-10)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3605468.3605498
Caspari LGreifenstein LHeuer UFraser GLaakso MMonga MSimon Sheard J(2023)ScratchLog: Live Learning Analytics for ScratchProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588836(403-409)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588836
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten