research-article

Is automated grading of models effective?: assessing automated grading of class diagrams

Authors:

Jörg KienzleAuthors Info & Claims

MODELS '20: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems

Pages 365 - 376

https://doi.org/10.1145/3365438.3410944

Published: 16 October 2020 Publication History

Abstract

Learning how to model the structural properties of a problem domain or an object-oriented design in the form of a class diagram is an essential learning task in many software engineering courses. Since the grading of models is a time-consuming activity, automated grading approaches have been developed to assist the instructor by speeding up the grading process, as well as ensuring consistency and fairness for large classrooms. This paper empirically evaluates the efficacy of one such automated grading approach when applied in two real world settings: a beginner undergraduate class of 103 students required to create an object-oriented design model, and an advanced undergraduate class of 89 students elaborating a domain model. The results of the experiment highlight a) the need to adapt the grading strategy and strictness to the level of the students and the grading style of the instructor, and b) the importance of considering multiple solution variants when grading. Modifications to the grading algorithm are proposed and validated experimentally.

References

[1]

2020. Coronavirus: Alberta universities look to mostly online courses for fall semester. https://globalnews.ca/news/6947098/coronavirus-alberta-universities-fall-semester/.

[2]

2020. Coursera. https://https://www.coursera.org/.

[3]

2020. Fall classes will mostly be done online amid COVID-19, some Canadian universities say. https://globalnews.ca/news/6935364/coronavirus-canadian-university-fall-classes/.

[4]

2020. How Online Learning Kept Higher Ed Open During the Coronavirus Crisis. https://spectrum.ieee.org/tech-talk/at-work/education/how-online-learning-kept-higher-ed-open-during-the-coronavirus-crisis.

[5]

2020. khanacademy. https://https://www.khanacademy.org/.

[6]

2020. Nine Ways To Reimagine Higher Education. https://www.forbes.com/sites/annkirschner/2020/05/14/nine-ways-to-reimagine-higher-education/#727f6c73767d.

[7]

2020. Udemy. https://https://www.udemy.org/.

[8]

Joel Adams. November 3, 2014. Computing Is The Safe STEM Career Choice Today. https://cacm.acm.org/blogs/blog-cacm/180053-computing-is-the-safe-stem-career-choice-today/fulltext

[9]

Rajeev Alur, Loris D'Antoni, Sumit Gulwani, Dileep Kini, and Mahesh Viswanathan. 2013. Automated grading of DFA constructions. In Twenty-Third International Joint Conference on Artificial Intelligence.

Digital Library

[10]

Mario Amelung, Peter Forbrig, and Dietmar Rösner. 2008. Towards generic and flexible web services for e-assessment. In Proceedings of the 13th annual conference on Innovation and technology in computer science education. 219--224.

Digital Library

[11]

Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment 4, 3 (2006).

[12]

Stephen P Balfour. 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review^TM. Research & Practice in Assessment 8 (2013), 40--48.

[13]

Firat Batmaz and Chris J Hinde. 2006. A diagram drawing tool for semi-automatic assessment of conceptual database diagrams. (2006).

[14]

Weiyi Bian, Omar Alam, and Jörg Kienzle. 2019. Automated Grading of Class Diagrams. In 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 700--709.

[15]

Liat Biberman-Shalev, Clara Sabbagh, Nura Resh, and Bracha Kramarski. 2011. Grading styles and disciplinary expertise: The mediating role of the teacher's perception of the subject matter. Teaching and Teacher Education 27, 5 (2011), 831--840.

[16]

Julio C Caiza and José María del ÁAlamo Ramiro. 2013. Programming assignments automatic grading: review of tools and implementations. (2013).

[17]

William Carbonaro. 2005. Tracking, students' effort, and academic achievement. Sociology of Education 78, 1 (2005), 27--49.

[18]

Janet Carter, Kirsti Ala-Mutka, Ursula Fuller, Martin Dick, John English, William Fone, and Judy Sheard. 2003. How shall we assess this? In Working group reports from ITiCSE on Innovation and technology in computer science education. 107--123.

[19]

Brenda Cheang, Andy Kurnia, Andrew Lim, and Wee-Chong Oon. 2003. On automated grading of programming assignments in an academic institution. Computers & Education 41, 2 (2003), 121--131.

Digital Library

[20]

Helder Pina Correia, José Paulo Leal, and Jose Carlos Paiva. 2017. Enhancing feedback to students in automated diagram assessment. (2017).

[21]

Thanasis Daradoumis, Roxana Bassi, Fatos Xhafa, and Santi Caballé. 2013. A review on massive e-learning (MOOC) design, delivery and assessment. In 2013 eighth international conference on P2P, parallel, grid, cloud and internet computing. IEEE, 208--213.

[22]

Maria De Marsico, Filippo Sciarrone, Andrea Sterbini, and Marco Temperini. 2017. Supporting mediated peer-evaluation to grade answers to open-ended questions. EURASIA J. Math. Sci. Technol. Educ 13, 4 (2017), 1085--1106.

[23]

Betty V DeBoer, Donna M Anderson, and Abdulaziz M Elfessi. 2007. Grading styles and instructor attitudes. College Teaching 55, 2 (2007), 57--64.

[24]

RL Doran, F Lawrenz, and S Helgeson. 1994. Research on assessment in science. Handbook ofresearch on science teaching and learning (1994), 388--442.

[25]

Joerg Evermann and Yair Wand. 2006. Ontological modeling rules for UML: An empirical assessment. Journal of Computer Information Systems 46, 5 (2006), 14--29.

[26]

Sue Fitzgerald, Brian Hanks, Raymond Lister, Renee McCauley, and Laurie Murphy. 2013. What Are We Thinking When We Grade Programs?. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE '13). Association for Computing Machinery, New York, NY, USA, 471--476.

Digital Library

[27]

Philippe Fournier-Viger, Roger Nkambou, and Engelbert Mephu Nguifo. 2010. Building Intelligent Tutoring Systems for Ill-Defined Domains. Springer Berlin Heidelberg, Berlin, Heidelberg, 81--101.

[28]

Jianxiong Gao, Bei Pang, and Steven S Lumetta. 2016. Automated feedback framework for introductory programming courses. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. 53--58.

Digital Library

[29]

Noraida Haji Ali, Zarina Shukur, and Sufian Idris. 2007. Assessment System For UML Class Diagram Using Notations Extraction. International Journal of Computer Science and Network Security 7, 8 (2007), 181--187.

[30]

Robert W. Hasker. 2011. UMLGrader: An Automated Class Diagram Grader. J. Comput. Sci. Coll. 27, 1 (Oct. 2011), 47--54.

[31]

Michael T Helmick. 2007. Interface-based programming assignments and automatic grading of java programs. In Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science education. 63--67.

Digital Library

[32]

Colin A Higgins, Geoffrey Gray, Pavlos Symeonidis, and Athanasios Tsintsifas. 2005. Automated assessment and experiences of teaching programming. Journal on Educational Resources in Computing (JERIC) 5, 3 (2005), 5-es.

Digital Library

[33]

Graeme Hirst, David St-Onge, et al. 1998. Lexical chains as representations of context for the detection and correction of malapropisms. WordNet: An electronic lexical database 305 (1998), 305--332.

[34]

Gil Hoggarth and Mike Lockyer. 1998. An Automated Student Diagram Assessment System. SIGCSE Bull. 30, 3 (Aug. 1998), 122--124.

Digital Library

[35]

James W Howatt. 1994. On criteria for grading student programs. ACM SIGCSE Bulletin 26, 3 (1994), 3--7.

Digital Library

[36]

RH Huang, DJ Liu, A Tlili, JF Yang, HH Wang, et al. 2020. Handbook on Facilitating Flexible Learning During Educational Disruption: The Chinese Experience in Maintaining Undisrupted Learning in COVID-19 Outbreak. Beijing: Smart Learning Institute of Beijing Normal University (2020).

[37]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10). ACM, New York, NY, USA, 86--93.

Digital Library

[38]

David Jackson and Michelle Usher. 1997. Grading student programs using ASSYST. In Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education. 335--339.

Digital Library

[39]

Shumin Jing, OC Santos, JG Boticario, C Romero, M Pechenizkiy, and A Merceron. 2015. Automatic Grading of Short Answers for MOOC via Semi-supervised Document Clustering. In EDM. 554--555.

[40]

JCS Kadupitiya, Surangika Ranathunga, and Gihan Dias. 2016. Automated assessment of multi-step answers for mathematical word problems. In 2016 Sixteenth International Conference on Advances in ICT for Emerging Regions (ICTer). IEEE, 66--71.

[41]

Mark Kassop. 2003. Ten ways online education matches, or surpasses, face-to-face learning. The Technology Source 3 (2003).

[42]

René F Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge. 170--179.

Digital Library

[43]

Andrew S Lan, Divyanshu Vats, Andrew E Waters, and Richard G Baraniuk. 2015. Mathematical language processing: Automatic grading and feedback for open response mathematical questions. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. 167--176.

Digital Library

[44]

Nguyen-Thinh Le, Frank Loll, and Niels Pinkwart. 2013. Operationalizing the Continuum Between Well-Defined and Ill-Defined Problems for Educational Technology. IEEE Trans. Learn. Technol. 6, 3 (July 2013), 258--270.

[45]

Chi Chiu Leung, Tai Hing Lam, and Kar Keung Cheng. 2020. Mass masking in the COVID-19 epidemic: people need guidance. Lancet 395, 10228 (2020), 945.

[46]

Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. 707--710.

[47]

Gerald S Leventhal. 1976. Fairness in social relationships. General Learning Press Morristown, NJ.

[48]

Dekang Lin. 1998. An Information-Theoretic Definition of Similarity. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML '98). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 296--304. http://dl.acm.org/citation.cfm?id=645527.657297

Digital Library

[49]

Michael Mohler and Rada Mihalcea. 2009. Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009). 567--575.

Digital Library

[50]

Paul L Nesbit and Suzan Burton. 2006. Student justice perceptions following assignment feedback. Assessment & Evaluation in Higher Education 31, 6 (2006), 655--670.

[51]

Vladimir L Pavlov and Anton Yatsenko. 2005. Using pantomime in teaching OOA&OOD with UML. In 18th Conference on Software Engineering Education & Training (CSEET'05). IEEE, 77--84.

Digital Library

[52]

Jennifer Randall and George Engelhard. 2010. Examining the grading practices of teachers. Teaching and Teacher Education 26, 7 (2010), 1372--1380.

[53]

Nura Resh. 2009. Justice in grades allocation: Teachers' perspective. Social Psychology of Education 12, 3 (2009), 315--325.

[54]

Shourya Roy, Sandipan Dandapat, Ajay Nagesh, and Yadati Narahari. 2016. Wisdom of students: A consistent automatic short answer grading technique. In Proceedings of the 13th International Conference on Natural Language Processing. 178--187.

[55]

Vasile Rus, Mihai Lintean, Rajendra Banjade, Nobal Bikram Niraula, and Dan Stefanescu. 2013. Semilar: The semantic similarity toolkit. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 163--168.

[56]

Jon Sáenz, Idoia G Gurtubay, Zunbeltz Izaola, and Gabriel A López. 2020. pygift-generator: A python module designed to prepare Moodle-based quizzes. arXiv preprint arXiv.2005.00910 (2020).

[57]

Kay Sambell, Sally Brown, and Liz McDowell. 1997. " But Is It Fair?": An Exploratory Study of Student Perceptions of the Consequential Validity of Assessment. Studies in educational evaluation 23, 4 (1997), 349--71.

[58]

Humasak Simanjuntak. 2015. Proposed framework for automatic grading system of ER diagram. In 2015 7th International Conference on Information Technology and Electrical Engineering (ICITEE). IEEE, 141--146.

[59]

Natasha Singer. January 24, 2019. The Hard Part of Computer Science? Getting Into Class. https://www.nytimes.com/2019/01/24/technology/computer-science-courses-college.html

[60]

J. Soler, I. Boada, F. Prados, J. Poch, and R. Fabregat. 2010. A web-based e-learning tool for UML class diagrams. In IEEE EDUCON 2010 Conference. 973--979.

[61]

Jaime Spacco, David Hovemeyer, William Pugh, Fawzi Emad, Jeffrey K Hollingsworth, and Nelson Padua-Perez. 2006. Experiences with marmoset: designing and using an advanced submission and testing system for programming courses. ACM Sigcse Bulletin 38,3 (2006), 13--17.

Digital Library

[62]

Dave R. Stikkolorum, Peter van der Putten, Caroline Sperandio, and Michel R. V. Chaudron. 2019. Towards Automated Grading of UML Class Diagrams with Machine Learning. In BNAIC/BENELEARN.

[63]

Michael Striewe and Michael Goedicke. 2014. Automated assessment of UML activity diagrams. In Proceedings of the 2014 conference on Innovation & technology in computer science education. 336--336.

Digital Library

[64]

Pete Thomas, Kevin Waugh, and Neil Smith. 2006. Using patterns in the automatic marking of ER-diagrams. In Proceedings of the 11th annual SIGCSE conference on Innovation and technology in computer science education. 83--87.

Digital Library

[65]

Pete Thomas, Kevin Waugh, and Neil Smith. 2008. A revision tool for teaching and learning sequence diagrams. In EdMedia+ Innovate Learning. Association for the Advancement of Computing in Education (AACE), 5454--5460.

[66]

Robin D Tierney. 2012. Fairness in classroom assessment. Sage (2012).

[67]

Robin D Tierney, Marielle Simon, and Julie Charland. 2011. Being fair: Teachers' interpretations of principles for standards-based grading. In The Educational Forum, Vol. 75. Taylor & Francis, 210--227.

[68]

Wim Van Lancker and Zachary Parolin. 2020. COVID-19, school closures, and child poverty: a social crisis in the making. The Lancet Public Health 5, 5 (2020), e243--e244.

[69]

Chuanyi Wang, Zhe Cheng, Xiao-Guang Yue, and Michael McAleer. 2020. Risk management of COVID-19 by universities in China.

[70]

Chris Wilcox. 2015. The role of automation in undergraduate computer science education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 90--95.

Digital Library

[71]

Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, 133--138.

Digital Library

Cited By

Sağlam THahner SSchmid LBurger ERoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Automated Detection of AI-Obfuscated Plagiarism in Modeling AssignmentsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640084(297-308)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640084
Alamr SLeon-Urrutia MCarr L(2023)E-assessment in Computer Science Higher EducationProceedings of the 15th International Conference on Education Technology and Computers10.1145/3629296.3629357(378-383)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3629296.3629357
Lawrence RFoss SUrazova T(2023)Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment SystemsACM Transactions on Computing Education10.1145/359121023:3(1-24)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3591210
Show More Cited By

Index Terms

Is automated grading of models effective?: assessing automated grading of class diagrams

Recommendations

Automated grading of class diagrams
MODELS '19: Proceedings of the 22nd International Conference on Model Driven Engineering Languages and Systems

Drawing UML diagrams, such as class diagrams, is an essential learning task in many software engineering courses. In course assignments, students are tasked to draw models that describe scenarios, model requirements, or system designs. The course ...
A framework for effectively utilising human grading input in automated short answer grading

Short answer questions are effective for recall knowledge assessment. Grading a large amount of short answers is costly and time consuming. To apply short answer questions on MOOCs platforms, the issues of scalability and responsiveness must be addressed. ...
Automated online grading for virtual machine-based systems administration courses
SIGCSE '13: Proceeding of the 44th ACM technical symposium on Computer science education

We present a system for automatically and iteratively grading student work in a Systems Administration course. This system can grade and give feedback regarding live (running) virtual machines the students have configured. It is appropriate for both ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MODELS '20: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems

October 2020

406 pages

ISBN:9781450370196

DOI:10.1145/3365438

General Chairs:
Eugene Syriani
Université de Montréal, Canada
,
Houari Sahraoui
Université de Montréal, Canada

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MODELS '20

Sponsor:

SIGSOFT

MODELS '20: ACM/IEEE 23rd International Conference on Model Driven Engineering Languages and Systems

October 16 - 23, 2020

Virtual Event, Canada

Acceptance Rates

MODELS '20 Paper Acceptance Rate 35 of 127 submissions, 28%;

Overall Acceptance Rate 118 of 382 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
314
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)10

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sağlam THahner SSchmid LBurger ERoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Automated Detection of AI-Obfuscated Plagiarism in Modeling AssignmentsProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640084(297-308)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640084
Alamr SLeon-Urrutia MCarr L(2023)E-assessment in Computer Science Higher EducationProceedings of the 15th International Conference on Education Technology and Computers10.1145/3629296.3629357(378-383)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3629296.3629357
Lawrence RFoss SUrazova T(2023)Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment SystemsACM Transactions on Computing Education10.1145/359121023:3(1-24)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3591210
Foss SUrazova TLawrence RKühn TSousa V(2022)Learning UML database design and modeling with AutoERProceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings10.1145/3550356.3559091(42-45)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3550356.3559091
Rukmono SChaudron MPéraire CSerebrenik A(2022)Guiding peer-feedback in learning software design using UMLProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3510456.3514148(122-133)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510456.3514148
Sychev OKamennov YMerkle LDoyle MSheard JSoh LDorn B(2022)Grading Mastery: Calculating Grades from Domain-Law ViolationsProceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 210.1145/3478432.3499135(1170-1170)Online publication date: 3-Mar-2022
https://dl.acm.org/doi/10.1145/3478432.3499135
Foss SUrazova TLawrence RMerkle LDoyle MSheard JSoh LDorn B(2022)Automatic Generation and Marking of UML Database Design DiagramsProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 110.1145/3478431.3499376(626-632)Online publication date: 22-Feb-2022
https://dl.acm.org/doi/10.1145/3478431.3499376

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents