Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Abstract
Artificial Intelligence-driven Development Environments (AIDEs) offer developers revolutionary computer programming assistance. There is great potential in incorporating AIDEs into Computer Science education; however, the effects of these tools should be fully examined before doing so. Here, a within-subjects study was conducted to compare the programming performance, workload, emotion, and self-efficacy of seventeen novices coding with and without use of the GitHub Copilot AIDE under time pressure. Results showed that using the AIDE significantly increased programming efficiency and reduced effort and mental workload but did not significantly impact emotion or self-efficacy. However, participants' performance improved with more experience using the AI, and their self-efficacy followed. The results suggest that students who try AIDEs will likely be tempted to use them for time-sensitive work. There is no evidence that providing AIDEs will aid struggling students, but there is a clear need for students to practice with AI to become competent and confident using it.
References
[1]
Naser Al Madi. 2023. How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE '22), January 05, 2023, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1--5. https://doi.org/10.1145/3551349.3560438
[2]
Naser Al Madi, Siyuan Peng, and Tamsin Rogers. 2022. Assessing Workload Perception in Introductory Computer Science Projects using NASA-TLX. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education, February 22, 2022, Providence RI USA. ACM, Providence RI USA, 668--674. https://doi.org/10.1145/3478431.3499406
[3]
Lecia J. Barker, Charlie McDowell, and Kimberly Kalahar. 2009. Exploring factors that influence computer science introductory course students to persist in the major. ACM SIGCSE Bull. 41, 1 (March 2009), 153--157. https://doi.org/10.1145/1539024.1508923
[4]
Ashok R. Basawapatna, Alexander Repenning, Kyu Han Koh, and Hilarie Nickerson. 2013. The zones of proximal flow: guiding students through a space of computational thinking skills and challenges. In Proceedings of the ninth annual international ACM conference on International computing education research, August 12, 2013, San Diego San California USA. ACM, San Diego San California USA, 67--74. https://doi.org/10.1145/2493394.2493404
[5]
Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, 2023. 500--506. .
[6]
Maureen Biggers, Anne Brauer, and Tuba Yilmaz. 2008. Student perceptions of computer science: a retention study comparing graduating seniors with cs leavers. ACM SIGCSE Bull. 40, 1 (March 2008), 402--406. https://doi.org/10.1145/1352322.1352274
[7]
Nigel Bosch and Sidney D'Mello. 2017. The Affective Experience of Novice Computer Programmers. Int. J. Artif. Intell. Educ. 27, 1 (March 2017), 181--206. https://doi.org/10.1007/s40593-015-0069--5
[8]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. ArXiv Prepr. ArXiv210703374 (2021).
[9]
Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic press.
[10]
Nasrin Dehbozorgi, Mary Lou Maher, and Mohsen Dorodchi. 2020. Sentiment analysis on conversations in collaborative active learning as an early predictor of performance. In 2020 IEEE Frontiers in Education Conference (FIE), 2020. IEEE, 1--9. Retrieved November 30, 2023 from https://ieeexplore.ieee.org/abstract/document/9274119/
[11]
James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (ACE '22), February 14, 2022, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 10--19. https://doi.org/10.1145/3511861.3511863
[12]
Peter Gerjets, Katharina Scheiter, and Richard Catrambone. 2006. Can learning from molar and modular worked examples be enhanced by providing instructional explanations and prompting self-explanations? Learn. Instr. 16, 2 (2006), 104--121.
[13]
Stefania Giannini. 2023. Reflections on generative AI and the future of education. Retrieved July 27, 2023 from https://unesdoc.unesco.org/ark:/48223/pf0000385877
[14]
GitHub. 2023. GitHub Copilot · Your AI pair programmer. GitHub. Retrieved July 24, 2023 from https://github.com/features/copilot
[15]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Elsevier, 139--183.
[16]
Toni Honicke and Jaclyn Broadbent. 2016. The influence of academic self-efficacy on academic performance: A systematic review. Educ. Res. Rev. 17, (2016), 63--84.
[17]
Torsten Hothorn, Kurt Hornik, Mark A. van de Wiel, Henric Winell, and Achim Zeileis. 2015. Conditional inference procedures in a permutation test framework. Coin Package R Version 31 3 Retrieved April 6, (2015), 2016.
[18]
IEEE. 2020. IEEE Recommended Practice for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-Being. IEEE Std 7010--2020 (May 2020), 1--96. https://doi.org/10.1109/IEEESTD.2020.9084219
[19]
Boris Iglewicz and David C. Hoaglin. 1993. Volume 16: how to detect and handle outliers. Quality Press.
[20]
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), April 19, 2023, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1--23. https://doi.org/10.1145/3544548.3580919
[21]
Paivi Kinnunen and Beth Simon. 2010. Experiencing programming assignments in CS1: the emotional toll. In Proceedings of the Sixth international workshop on Computing education research, August 09, 2010, Aarhus Denmark. ACM, Aarhus Denmark, 77--86. https://doi.org/10.1145/1839594.1839609
[22]
Laerd Statistics. 2015. Tutorials for SPSS Statistics. Retrieved December 12, 2023 from https://statistics.laerd.com/
[23]
Mike LePine. 2022. Chapter 9.4: Distribution Needed for Hypothesis Testing. In College Statistics. St. Clair College AA&T. Retrieved December 1, 2023 from https://ecampusontario.pressbooks.pub/sccstatistics/chapter/distribution-needed-for-hypothesis-testing/
[24]
Alex Lishinski and Joshua Rosenberg. 2021. All the Pieces Matter: The Relationship of Momentary Self-efficacy and Affective Experiences with CS1 Achievement and Interest in Computing. In Proceedings of the 17th ACM Conference on International Computing Education Research, August 16, 2021, Virtual Event USA. ACM, Virtual Event USA, 252--265. https://doi.org/10.1145/3446871.3469740
[25]
Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. In Working group reports from ITiCSE on Innovation and technology in computer science education (ITiCSE-WGR '01), December 01, 2001, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 125--180. https://doi.org/10.1145/572133.572137
[26]
Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot's code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR '22), October 17, 2022, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1--5. https://doi.org/10.1145/3524842.3528470
[27]
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. ArXiv Prepr. ArXiv220313474 (2022).
[28]
Julie Pallant. 2007. SPSS Survival Manual. Open University Press New York, NY, USA.
[29]
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), 2022. IEEE, 754--768. .
[30]
Andrew Petersen, Michelle Craig, Jennifer Campbell, and Anya Tafliovich. 2016. Revisiting why students drop CS1. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research, November 24, 2016, Koli Finland. ACM, Koli Finland, 71--80. https://doi.org/10.1145/2999541.2999552
[31]
Matei-Dan Popovici. 2023. ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course. Int. J. Hum.-Comput. Interact. (October 2023). https://doi.org/10.1080/10447318.2023.2269006
[32]
James A. Russell. 1980. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 6 (1980), 1161.
[33]
John Sweller. 2011. Cognitive load theory. In Psychology of learning and motivation. Elsevier, 37--76. Retrieved December 28, 2023 from https://www.sciencedirect.com/science/article/pii/B9780123876911000028
[34]
Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA '22), April 28, 2022, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/3491101.3519665
[35]
Kimberly Wilson and Anupama Narayan. 2016. Relationships among individual task self-efficacy, self-regulated learning strategy use and academic performance in a computer-supported collaborative learning environment. Educ. Psychol. 36, 2 (February 2016), 236--253. https://doi.org/10.1080/01443410.2014.926312
[36]
Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Trans. Softw. Eng. Methodol. 31, 2 (March 2022), 29:1--29:47. https://doi.org/10.1145/3487569
[37]
Frank Yates. 1934. Contingency tables involving small numbers and the X 2 test. Suppl. J. R. Stat. Soc. 1, 2 (1934), 217--235.
[38]
Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot's code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, 2022. 62--71. .
Index Terms
- Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Recommendations
Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1We explored how undergraduate introductory programming students naturalistically used generative AI to solve programming problems. We focused on the relationship between their use of AI to their self-regulation strategies, self-efficacy, and fear of ...
Going SOLO to assess novice programmers
ITiCSE '08: Proceedings of the 13th annual conference on Innovation and technology in computer science educationThis paper explores the programming knowledge of novices using Biggs' SOLO taxonomy. It builds on previous work of Lister et al. (2006) and addresses some of the criticisms of that work. The research was conducted by studying the exam scripts for 120 ...
Going SOLO to assess novice programmers
ITiCSE '08This paper explores the programming knowledge of novices using Biggs' SOLO taxonomy. It builds on previous work of Lister et al. (2006) and addresses some of the criticisms of that work. The research was conducted by studying the exam scripts for 120 ...
Comments
Information & Contributors
Information
Published In
July 2024
776 pages
ISBN:9798400706004
DOI:10.1145/3649217
- General Chairs:
- Mattia Monga,
- Violetta Lonati,
- Erik Barendsen,
- Program Chairs:
- Judithe Sheard,
- James Paterson
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Commonwealth Cyber Initiative
- National Science Foundation
Conference
ITiCSE 2024
Sponsor:
Acceptance Rates
Overall Acceptance Rate 552 of 1,613 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 370Total Downloads
- Downloads (Last 12 months)370
- Downloads (Last 6 weeks)96
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in