Automated Assessment: Does It Align With Teachers' Views?
Authors: 
Henry Hickman, Tim BellAuthors Info & Claims
Article No.: 10, Pages 1 - 10
Abstract
In Aotearoa New Zealand, assessment of programming for the national NCEA standards is carried out manually by teachers, many of whom are not experienced programmers. In an attempt to decrease teacher workload, we have adapted Moodle CodeRunner [23] to assess a widely used recently released high school programming standard. This paper explores in detail how we have automated each criterion of the new standard, including dealing with judgement calls for the more subjective criteria.
We then report on interviews with experienced programming teachers who were shown example tasks from our system, as well as model answers for each example. We found that teachers were enthusiastic about using automated assessment to assess the standard, and while there wasn’t one agreed upon interpretation of the standard, teachers were happy with how the system supported marking. We also found no universal agreement among the level of context desired in programming questions to assess the standard, despite the small sample size. These interviews have given us confidence to both release these examples to teachers across New Zealand, many of whom are struggling with how to teach the standard, and to move forward with the pilot of our automated assessment system.
References
[1]
Hirohisa Aman, Sousuke Amasaki, Tomoyuki Yokogawa, and Minoru Kawahara. 2021. A Large-Scale Investigation of Local Variable Names in Java Programs: Is Longer Name Better for Broader Scope Variable?. In Quality of Information and Communications Technology, Ana C. R. Paiva, Ana Rosa Cavalli, Paula Ventura Martins, and Ricardo Pérez-Castillo (Eds.). Springer International Publishing, Cham, 489–500.
[2]
Tim Bell, Peter Andreae, and Anthony Robins. 2012. Computer Science in NZ High Schools: The First Year of the New Standards. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education(SIGCSE ’12). Association for Computing Machinery, New York, NY, USA, 343–348. https://doi.org/10.1145/2157136.2157240 event-place: Raleigh, North Carolina, USA.
[3]
Tim Bell, Heidi Newton, Peter Andreae, and Anthony Robins. 2012. The Introduction of Computer Science to NZ High Schools: An Analysis of Student Work. In Proceedings of the 7th Workshop in Primary and Secondary Computing Education(WiPSCE ’12). Association for Computing Machinery, New York, NY, USA, 5–15. https://doi.org/10.1145/2481449.2481454 event-place: Hamburg, Germany.
[4]
Neil C. C. Brown and Amjad Altadmri. 2017. Novice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs. ACM Trans. Comput. Educ. 17, 2, Article 7 (may 2017), 21 pages. https://doi.org/10.1145/2994154
[5]
Don Coleman, Dan Ash, Bruce Lowther, and Paul Oman. 1994. Using metrics to evaluate software system maintainability. Computer 27, 8 (1994), 44–49. Publisher: IEEE.
[6]
Liane Colonna. 2023. Teachers in the loop? An analysis of automatic assessment systems under Article 22 GDPR. International Data Privacy Law 14, 1 (Nov. 2023), 3–18. https://doi.org/10.1093/idpl/ipad024 _eprint: https://academic.oup.com/idpl/article-pdf/14/1/3/57067439/ipad024.pdf.
[7]
Michelle Craig, Jacqueline Smith, and Andrew Petersen. 2017. Familiar contexts and the difficulty of programming problems. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research(Koli Calling ’17). Association for Computing Machinery, New York, NY, USA, 123–127. https://doi.org/10.1145/3141880.3141898 event-place: Koli, Finland.
[8]
Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia M. de Oliveira. 2005. A study of the documentation essential to software maintenance. In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information(SIGDOC ’05). Association for Computing Machinery, New York, NY, USA, 68–75. https://doi.org/10.1145/1085313.1085331 event-place: Coventry, United Kingdom.
[9]
Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia M. de Oliveira. 2005. A Study of the Documentation Essential to Software Maintenance. In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information (Coventry, United Kingdom) (SIGDOC ’05). Association for Computing Machinery, New York, NY, USA, 68–75. https://doi.org/10.1145/1085313.1085331
[10]
Anne De Roeck Debra Trusso Haley, Pete Thomas and Marian Petre. 2007. Seeing the Whole Picture: Evaluating Automated Assessment Systems. Innovation in Teaching and Learning in Information and Computer Sciences 6, 4 (2007), 203–224. https://doi.org/10.11120/ital.2007.06040203 Publisher: Routledge _eprint: https://doi.org/10.11120/ital.2007.06040203.
[11]
Paul Denny, Sathiamoorthy Manoharan, Ulrich Speidel, Giovanni Russello, and Angela Chang. 2019. On the Fairness of Multiple-Variant Multiple-Choice Examinations. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 462–468. https://doi.org/10.1145/3287324.3287357
[12]
Dror G Feitelson, Ayelet Mizrahi, Nofar Noy, Aviad Ben Shabat, Or Eliyahu, and Roy Sheffer. 2020. How developers choose names. IEEE Transactions on Software Engineering 48, 1 (2020), 37–52. Publisher: IEEE.
[13]
James Finnie-Ansley, Paul Denny, and Andrew Luxton-Reilly. 2021. A Semblance of Similarity: Student Categorisation of Simple Algorithmic Problem Statements. In Proceedings of the 17th ACM Conference on International Computing Education Research(ICER 2021). Association for Computing Machinery, New York, NY, USA, 198–212. https://doi.org/10.1145/3446871.3469745 event-place: Virtual Event, USA.
[14]
Sue Fitzgerald, Brian Hanks, Raymond Lister, Renee McCauley, and Laurie Murphy. 2013. What Are We Thinking When We Grade Programs?. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 471–476. https://doi.org/10.1145/2445196.2445339
[15]
Richard A Griggs and James R Cox. 1982. The elusive thematic-materials effect in Wason’s selection task. British journal of psychology 73, 3 (1982), 407–420. Publisher: Wiley Online Library.
[16]
Marcelo Guerra Hahn, Silvia Margarita Baldiris Navarro, Luis De La Fuente Valentín, and Daniel Burgos. 2021. A Systematic Review of the Effects of Automatic Scoring and Automatic Feedback in Educational Settings. IEEE Access 9 (2021), 108190–108198. https://doi.org/10.1109/ACCESS.2021.3100890
[17]
Ayman Hajja, Austin J. Hunt, and Renée McCauley. 2019. PolyPy: A Web-Platform for Generating Quasi-Random Python Code and Gaining Insights on Student Learning. In 2019 IEEE Frontiers in Education Conference (FIE). 1–8. https://doi.org/10.1109/FIE43999.2019.9028712
[18]
Emlyn Hegarty-Kelly and Dr Aidan Mooney. 2021. Analysis of an Automatic Grading System within First Year Computer Science Programming Modules. In Computing Education Practice 2021(CEP ’21). Association for Computing Machinery, New York, NY, USA, 17–20. https://doi.org/10.1145/3437914.3437973 event-place: Durham, United Kingdom.
[19]
Henry Hickman and Tim Bell. 2023. Towards Automated Assessment of High School Programming. In 2023 IEEE Frontiers in Education Conference.
[20]
Anders Jonsson. 2014. Rubrics as a way of providing transparency in assessment. Assessment & Evaluation in Higher Education 39, 7 (2014), 840–852. https://doi.org/10.1080/02602938.2013.875117 arXiv:https://doi.org/10.1080/02602938.2013.875117
[21]
Hassan Khosravi, Simon Buckingham Shum, Guanliang Chen, Cristina Conati, Yi-Shan Tsai, Judy Kay, Simon Knight, Roberto Martinez-Maldonado, Shazia Sadiq, and Dragan Gašević. 2022. Explainable Artificial Intelligence in education. Computers and Education: Artificial Intelligence 3 (2022), 100074. https://doi.org/10.1016/j.caeai.2022.100074
[22]
Noela Jemutai Kipyegen and William P K Korir. 2013. Importance of Software Documentation. International Journal of Computer Science Issues (IJCSI) 10, 5 (Sept. 2013), 223–228. https://www.proquest.com/scholarly-journals/importance-software-documentation/docview/1477204708/se-2?accountid=14499 Place: Mahebourg Publisher: International Journal of Computer Science Issues (IJCSI).
[23]
Richard Lobb and Jenny Harlow. 2016. Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads 7, 1 (Feb. 2016), 47–51. https://doi.org/10.1145/2810041 Place: New York, NY, USA Publisher: Association for Computing Machinery.
[24]
Ellie Lovellette, Dennis J Bouvier, and John Matta. 2024. Contextualization, Authenticity, and the Problem Description Effect. ACM Trans. Comput. Educ. 24, 2 (April 2024). https://doi.org/10.1145/3643864 Place: New York, NY, USA Publisher: Association for Computing Machinery.
[25]
Ellie Lovellette, John Matta, Dennis Bouvier, and Roger Frye. 2017. Just the Numbers: An Investigation of Contextualization of Problems for Novice Programmers. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’17). Association for Computing Machinery, New York, NY, USA, 393–398. https://doi.org/10.1145/3017680.3017726 event-place: Seattle, Washington, USA.
[26]
T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering SE-2, 4 (1976), 308–320. https://doi.org/10.1109/TSE.1976.233837
[27]
Marcus Messer, Neil C. C. Brown, Michael Kölling, and Miaojing Shi. 2024. Automated Grading and Feedback Tools for Programming Education: A Systematic Review. ACM Trans. Comput. Educ. 24, 1 (Feb. 2024). https://doi.org/10.1145/3636515 Place: New York, NY, USA Publisher: Association for Computing Machinery.
[28]
Daphne Miedema, Toni Taipalus, and Efthimia Aivaloglou. 2023. Students’ Perceptions on Engaging Database Domains and Structures. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1(SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 122–128. https://doi.org/10.1145/3545945.3569727 event-place:, Toronto ON, Canada,.
[29]
NCEA. 2022. Create a computer program. https://ncea.education.govt.nz/technology/digital-technologies/1/1?view=standard/. Accessed: August 11 2022.
[30]
NCEA. 2022. The new home for NCEA. https://ncea.education.govt.nz/. Accessed: August 12 2022.
[31]
James Perretta, Westley Weimer, and Andrew DeOrio. 2019. Human vs. automated coding style grading in computing education. In 2019 ASEE Annual Conference & Exposition.
[32]
Ashwin Rachha and Mohammed Seyam. 2023. Explainable AI In Education : Current Trends, Challenges, And Opportunities. In SoutheastCon 2023. 232–239. https://doi.org/10.1109/SoutheastCon51012.2023.10115140
[33]
Md Mostafizer Rahman and Yutaka Watanobe. 2023. Chatgpt for education and research: Opportunities, threats, and strategies. Applied Sciences 13, 9 (2023), 5783. Publisher: MDPI.
[34]
Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How do professional developers comprehend software?. In 2012 34th International Conference on Software Engineering (ICSE). 255–265. https://doi.org/10.1109/ICSE.2012.6227188
[35]
Shwetlena Sabarwal, Malek Abu-Jawdeh, and Radhika Kapoor. 2021. Teacher Beliefs: Why They Matter and What They Are. The World Bank Research Observer 37, 1 (June 2021), 73–106. https://doi.org/10.1093/wbro/lkab008 _eprint: https://academic.oup.com/wbro/article-pdf/37/1/73/42793384/lkab008.pdf.
[36]
Chamindi K. Samarasekara, Claudia Ott, and Anthony Robins. 2021. A Decade of CS Education in New Zealand’s High Schools: Where Are We At? In The 16th Workshop in Primary and Secondary Computing Education. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3481312.3481321
[37]
Chamindi K. Samarasekara, Claudia Ott, and Anthony Robins. 2022. Barriers to New Zealand High School CS Education - Learners’ Perspectives. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1(SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 927–933. https://doi.org/10.1145/3478431.3499344 event-place: Providence, RI, USA.
[38]
Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering SE-10, 5 (1984), 595–609. https://doi.org/10.1109/TSE.1984.5010283
[39]
M Taylor, E Cawley, S De Jesus, M Stehlik, and D Kosbie. 2022. CMU CS academy: a test-based programming language approach for High School Computer Science Education. In EDULEARN22 Proceedings. IATED, 9091–9100.
[40]
UC Computer Science Education. 2021. codeWOF. https://www.codewof.co.nz/. Accessed: October 13 2022.
[41]
Eva Van de Weijer-Bergsma and Sanne H. G. Van der Ven. 2021. Why and for whom does personalizing math problems enhance performance? Testing the mediation of enjoyment and cognitive load at different ability levels. Learning and Individual Differences 87 (2021), 101982. https://doi.org/10.1016/j.lindif.2021.101982
[42]
Chris Wilcox. 2015. The Role of Automation in Undergraduate Computer Science Education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education(SIGCSE ’15). Association for Computing Machinery, New York, NY, USA, 90–95. https://doi.org/10.1145/2676723.2677226 event-place: Kansas City, Missouri, USA.
[43]
Soundous Zougari, Mariam Tanana, and Abdelouahid Lyhyaoui. 2022. Validity of a graph-based automatic assessment system for programming assignments: human versus automatic grading. International Journal of Electrical and Computer Engineering (IJECE) 12 (06 2022), 2867. https://doi.org/10.11591/ijece.v12i3.pp2867-2875
Index Terms
- Automated Assessment: Does It Align With Teachers' Views?
Recommendations
A survey of Australian teachers' self-efficacy and assessment approaches for the K-12 digital technologies curriculum
WiPSCE '18: Proceedings of the 13th Workshop in Primary and Secondary Computing EducationAs K-12 computer science (CS) education has been introduced to a number of countries around the world, the CS education community has been busy working to understand the learning and teaching of computing at these year levels, as well as how to build ...
Investigating Learners’ Views of Assessment Types in Massive Open Online Courses (MOOCs)
Design for Teaching and Learning in a Networked WorldAbstractMassive Open Online Courses (MOOCs) are changing the contours of the teaching and learning landscape. Assessment covers an important part of this landscape and may be a key driver for learning. This paper presents preliminary results of a ...
Adoption of new computer science high school standards by New Zealand teachers
WiPSE '13: Proceedings of the 8th Workshop in Primary and Secondary Computing EducationComputer science was progressively introduced as a subject in New Zealand high schools between 2011 and 2013, and teachers have played a key role in the success of the introduction of the new curriculum. This paper looks at how the process has worked ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2024
203 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
WiPSCE '24
WiPSCE '24: The 19th WiPSCE Conference on Primary and Secondary Computing Education Research
September 16 - 18, 2024
Munich, Germany
Acceptance Rates
Overall Acceptance Rate 104 of 279 submissions, 37%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 80Total Downloads
- Downloads (Last 12 months)80
- Downloads (Last 6 weeks)47
Reflects downloads up to 23 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in