research-article

Open access

“It's not like Jarvis, but it's pretty close!” - Examining ChatGPT's Usage among Undergraduate Students in Computer Science

Authors:

Ritvik Budhiraja,

Jagat Sesh Challa,

Harshal D. Akolekar,

Dhruv KumarAuthors Info & Claims

ACE '24: Proceedings of the 26th Australasian Computing Education Conference

Pages 124 - 133

https://doi.org/10.1145/3636243.3636257

Published: 29 January 2024 Publication History

All formats PDF

Abstract

Large language models (LLMs) such as ChatGPT and Google Bard have garnered significant attention in the academic community. Previous research has evaluated these LLMs for various applications such as generating programming exercises and solutions. However, these evaluations have predominantly been conducted by instructors and researchers, not considering the actual usage of LLMs by students. This study adopts a student-first approach to comprehensively understand how undergraduate computer science students utilize ChatGPT, a popular LLM, released by OpenAI. We employ a combination of student surveys and interviews to obtain valuable insights into the benefits, challenges, and suggested improvements related to ChatGPT. Our findings suggest that a majority of students (over 57%) have a convincingly positive outlook towards adopting ChatGPT as an aid in coursework-related tasks. However, our research also highlights various challenges that must be resolved for long-term acceptance of ChatGPT amongst students. The findings from this investigation have broader implications and may be applicable to other LLMs and their role in computing education.

References

[1]

[n. d.]. AlphaCode. https://alphacode.deepmind.com/. Accessed: 2023-08-18.

[2]

[n. d.]. Amazon CodeWhisperer. https://aws.amazon.com/codewhisperer/. Accessed: 2023-08-18.

[3]

[n. d.]. Bing. https://www.bing.com/. Accessed: 2023-08-18.

[4]

[n. d.]. Chat GPT. https://chat.openai.com/. Accessed: 2023-08-18.

[5]

[n. d.]. Github Copilot. https://github.com/features/copilot. Accessed: 2023-08-18.

[6]

[n. d.]. Google Bard. https://bard.google.com/. Accessed: 2023-08-18.

[7]

[n. d.]. GPT-3. https://openai.com/blog/gpt-3-apps. Accessed: 2023-08-18.

[8]

[n. d.]. Mixed Methods Approach. https://en.wikipedia.org/wiki/Multimethodology. Accessed: 2023-08-18.

[9]

[n. d.]. OpenAI Codex. https://openai.com/blog/openai-codex. Accessed: 2023-08-18.

[10]

[n. d.]. Snowball sampling. https://en.wikipedia.org/wiki/Snowball_sampling. Accessed: 2023-08-18.

[11]

[n. d.]. Stack Overflow. https://stackoverflow.com/. Accessed: 2023-08-18.

[12]

Devi Akella. 2010. Learning together: Kolb’s experiential theory and its application. Journal of Management Organization - J MANAG ORGAN 16 (March 2010), 100–112. https://doi.org/10.5172/jmo.16.1.100

[13]

Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023. Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 292–298. https://doi.org/10.1145/3587102.3588852

Digital Library

[14]

Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 500–506. https://doi.org/10.1145/3545945.3569759

Digital Library

[15]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (Jan. 2006), 77–101. https://doi.org/10.1191/1478088706qp063oa

[16]

Bruno Pereira Cipriano and Pedro Alves. 2023. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In Proceedings of the 2023 ITiCSE Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 61–67. https://doi.org/10.1145/3587102.3588814

Digital Library

[17]

Marian Daun and Jennifer Brings. 2023. How ChatGPT Will Change Software Engineering Education. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 110–116. https://doi.org/10.1145/3587102.3588815

Digital Library

[18]

Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 1136–1142. https://doi.org/10.1145/3545945.3569823

Digital Library

[19]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A Survey on In-context Learning. https://doi.org/10.48550/arXiv.2301.00234 arXiv:2301.00234 [cs].

[20]

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). ACM, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863

Digital Library

[21]

James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th ACE Conference (Melbourne, VIC, Australia) (ACE ’23). ACM, New York, NY, USA, 97–104. https://doi.org/10.1145/3576123.3576134

Digital Library

[22]

Mehmet Firat. 2023. What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning & Teaching 6, 1 (2023), 1–22. https://doi.org/10.37074/jalt.2023.6.1.22

[23]

Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30 (2020), 681–694.

Digital Library

[24]

Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xin Zhou. 2023. Better Zero-Shot Reasoning with Role-Play Prompting. https://doi.org/10.48550/arXiv.2308.07702 arXiv:2308.07702 [cs].

[25]

Lisa Lacher and Cydnee Biehl. 2018. Using Discord to Understand and Moderate Collaboration and Teamwork: (Abstract Only). In Proceedings of the 49th ACM Technical Symposium on Computer Science Education(SIGCSE ’18). Association for Computing Machinery, New York, NY, USA, 1107. https://doi.org/10.1145/3159450.3162231

Digital Library

[26]

Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 124–130. https://doi.org/10.1145/3587102.3588785

Digital Library

[27]

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 563–569. https://doi.org/10.1145/3545945.3569770

Digital Library

[28]

Abe Leite and Saúl A. Blanco. 2020. Effects of Human vs. Automatic Feedback on Students’ Understanding of AI Concepts and Programming Style. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education(SIGCSE ’20). ACM, New York, NY, USA, 44–50. https://doi.org/10.1145/3328778.3366921

Digital Library

[29]

Xinyue Lin, James Connors, Chang Lim, and John R. Hott. 2021. How Do Students Collaborate? Analyzing Group Choice in a Collaborative Learning Environment. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education(SIGCSE ’21). ACM, New York, NY, USA, 212–218. https://doi.org/10.1145/3408877.3432389

Digital Library

[30]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. https://doi.org/10.48550/arXiv.2304.08485 arXiv:2304.08485 [cs].

[31]

Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 931–937. https://doi.org/10.1145/3545945.3569785

Digital Library

[32]

Kamil Malinka, Martin Peresíni, Anton Firc, Ondrej Hujnák, and Filip Janus. 2023. On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 47–53. https://doi.org/10.1145/3587102.3588827

Digital Library

[33]

Amy Miner, Jennifer Mallow, Laurie Theeke, and Emily Barnes. 2015. Using Gagne’s 9 Events of Instruction to Enhance Student Performance and Course Evaluations in Undergraduate Nursing Course. Nurse educator 40, 3 (2015), 152–154. https://doi.org/10.1097/NNE.0000000000000138

[34]

OpenAI. 2016. Generative models. https://openai.com/research/generative-models

[35]

Eng Lieh Ouh, Benjamin Kok Siew Gan, Kyong Jin Shim, and Swavek Wlodkowski. 2023. ChatGPT, Can You Generate Solutions for My Coding Exercises? An Evaluation on Its Effectiveness in an Undergraduate Java Programming Course. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 54–60. https://doi.org/10.1145/3587102.3588794

Digital Library

[36]

Amy Rechkemmer and Ming Yin. 2022. When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). ACM, New York, NY, USA, 1–14. https://doi.org/10.1145/3491102.3501967

Digital Library

[37]

Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the ITiCSE 2023 Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 299–305. https://doi.org/10.1145/3587102.3588805

Digital Library

[38]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER ’22). ACM, New York, NY, USA, 27–43. https://doi.org/10.1145/3501385.3543957

Digital Library

[39]

Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-Trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?. In Proceedings of the 2023 ITiCSE Conference V. 1 (Turku, Finland) (ITiCSE 2023). ACM, New York, NY, USA, 117–123. https://doi.org/10.1145/3587102.3588792

Digital Library

[40]

Abdulhadi Shoufan. 2023. Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up Survey. IEEE Access 11 (2023), 38805–38818. https://doi.org/10.1109/ACCESS.2023.3268224

[41]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs].

[42]

Michel Wermelinger. 2023. Using GitHub Copilot to Solve Simple Programming Problems. In Proceedings of the 54th ACM SIGCSE TS V. 1 (Toronto ON, Canada) (SIGCSE 2023). ACM, New York, NY, USA, 172–178. https://doi.org/10.1145/3545945.3569830

Digital Library

[43]

Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. https://doi.org/10.48550/arXiv.2302.11382 arXiv:2302.11382 [cs].

Cited By

Maguire JEnglish RCao QSeow C(2025)Themes in the Declared Use of Generative Artificial Intelligence in AssessmentProceedings of the 9th Conference on Computing Education Practice10.1145/3702212.3702217(17-20)Online publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1145/3702212.3702217
Garcia M(2025)Teaching and learning computer programming using ChatGPT: A rapid review of literature amid the rise of generative AI technologiesEducation and Information Technologies10.1007/s10639-025-13452-5Online publication date: 1-Mar-2025
https://doi.org/10.1007/s10639-025-13452-5
Humble N(2024)Risk management strategy for generative AI in computing education: how to handle the strengths, weaknesses, opportunities, and threats?International Journal of Educational Technology in Higher Education10.1186/s41239-024-00494-x21:1Online publication date: 11-Dec-2024
https://doi.org/10.1186/s41239-024-00494-x
Show More Cited By

Index Terms

“It's not like Jarvis, but it's pretty close!” - Examining ChatGPT's Usage among Undergraduate Students in Computer Science

Recommendations

ACM Undergraduate Curricular Guidance in Computer Science: The First Two Years
ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

Under the auspices of the ACM Education Board, the Committee for Computing Education in Community Colleges (CCECC) is updating the 2009 ACM associate-degree curricular guidance in computer science with inclusion of contemporary cybersecurity concepts. ...
Understanding How Computer Science Undergraduate Students are Developing their Professional Identities: (Abstract Only)
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

Understanding the development of professional identity in Computer Science (CS) undergraduate students can help better evaluate CS degree program's effectiveness in preparing students for their career goals. This poster presents findings from a study ...
Understanding Professional Identities and Goals of Computer Science Undergraduate Students
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

Understanding professional goals and identities of undergraduate Computer Science (CS) students is critical for curriculum decisions, workforce development, and retention programs. This paper aims to explore the ways in which undergraduate CS students ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACE '24: Proceedings of the 26th Australasian Computing Education Conference

January 2024

208 pages

ISBN:9798400716195

DOI:10.1145/3636243

Editors:
Nicole Herbert
University of Tasmania
,
Carolyn Seton
Southern Cross University

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 January 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ACE 2024

ACE 2024: Australian Computing Education Conference

January 29 - February 2, 2024

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 161 of 359 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
862
Total Downloads

Downloads (Last 12 months)795
Downloads (Last 6 weeks)99

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Maguire JEnglish RCao QSeow C(2025)Themes in the Declared Use of Generative Artificial Intelligence in AssessmentProceedings of the 9th Conference on Computing Education Practice10.1145/3702212.3702217(17-20)Online publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1145/3702212.3702217
Garcia M(2025)Teaching and learning computer programming using ChatGPT: A rapid review of literature amid the rise of generative AI technologiesEducation and Information Technologies10.1007/s10639-025-13452-5Online publication date: 1-Mar-2025
https://doi.org/10.1007/s10639-025-13452-5
Humble N(2024)Risk management strategy for generative AI in computing education: how to handle the strengths, weaknesses, opportunities, and threats?International Journal of Educational Technology in Higher Education10.1186/s41239-024-00494-x21:1Online publication date: 11-Dec-2024
https://doi.org/10.1186/s41239-024-00494-x
Keuning HAlpizar-Chacon ILykourentzou IBeehler LKöppe Cde Jong ISosnovsky S(2024)Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing CoursesProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699546(1-12)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699546
Stone I(2024)Exploring Human-Centered Approaches in Generative AI and Introductory Programming Research: A Scoping ReviewProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689553(1-7)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3689535.3689553
Stone I(2024)Investigating the Use of ChatGPT to Support the Learning of Python Programming Among Upper Secondary School Students: A Design-Based Research StudyProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689537(1-1)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3689535.3689537
Adnin RDas M(2024)"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI ToolsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675631(1-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675631
Grande VKiesler NFrancisco R. MMonga MLonati VBarendsen ESheard JPaterson J(2024)Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional EthicsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653624(478-484)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653624
Kim NKo HMyers GBach B(2024)ChatGPT in Data Visualization Education: A Student Perspective2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00022(109-120)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00022

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten