Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education

Nischal Ashok Kumar, Andrew S. Lan
Proceedings of the 2024 AAAI Conference on Artificial Intelligence, PMLR 257:170-179, 2024.

Abstract

In computer science education, test cases are an integral part of programming assignments since they can be used as assessment items to test students' programming knowledgeand provide personalized feedback on student-written code. The goal of our work is to propose a fully automated approach for test case generation that can accurately measure student knowledge, which is important for two reasons. First, manually constructing testcases requires expert knowledge and is a labor-intensive process. Second, developing test cases for students, especially those who are novice programmers, is significantly different from those oriented toward professional-level software developers. Therefore, we need an automated process for test case generation to assess student knowledge and provide feedback. In this work, we propose a large language model-based approach to automatically generate test cases and show that they are good measures of student knowledge, using a publicly available dataset that contains student-written Java code. We also discuss future research directions centered on using test cases to help students.

Cite this Paper

BibTeX
@InProceedings{pmlr-v257-kumar24, title = {Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education}, author = {Ashok Kumar, Nischal and Andrew S., Lan}, booktitle = {Proceedings of the 2024 AAAI Conference on Artificial Intelligence}, pages = {170--179}, year = {2024}, editor = {Ananda, Muktha and Malick, Debshila Basu and Burstein, Jill and Liu, Lydia T. and Liu, Zitao and Sharpnack, James and Wang, Zichao and Wang, Serena}, volume = {257}, series = {Proceedings of Machine Learning Research}, month = {26--27 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v257/main/assets/kumar24/kumar24.pdf}, url = {https://proceedings.mlr.press/v257/kumar24.html}, abstract = {In computer science education, test cases are an integral part of programming assignments since they can be used as assessment items to test students' programming knowledgeand provide personalized feedback on student-written code. The goal of our work is to propose a fully automated approach for test case generation that can accurately measure student knowledge, which is important for two reasons. First, manually constructing testcases requires expert knowledge and is a labor-intensive process. Second, developing test cases for students, especially those who are novice programmers, is significantly different from those oriented toward professional-level software developers. Therefore, we need an automated process for test case generation to assess student knowledge and provide feedback. In this work, we propose a large language model-based approach to automatically generate test cases and show that they are good measures of student knowledge, using a publicly available dataset that contains student-written Java code. We also discuss future research directions centered on using test cases to help students.} }
Endnote
%0 Conference Paper %T Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education %A Nischal Ashok Kumar %A Andrew S. Lan %B Proceedings of the 2024 AAAI Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Muktha Ananda %E Debshila Basu Malick %E Jill Burstein %E Lydia T. Liu %E Zitao Liu %E James Sharpnack %E Zichao Wang %E Serena Wang %F pmlr-v257-kumar24 %I PMLR %P 170--179 %U https://proceedings.mlr.press/v257/kumar24.html %V 257 %X In computer science education, test cases are an integral part of programming assignments since they can be used as assessment items to test students' programming knowledgeand provide personalized feedback on student-written code. The goal of our work is to propose a fully automated approach for test case generation that can accurately measure student knowledge, which is important for two reasons. First, manually constructing testcases requires expert knowledge and is a labor-intensive process. Second, developing test cases for students, especially those who are novice programmers, is significantly different from those oriented toward professional-level software developers. Therefore, we need an automated process for test case generation to assess student knowledge and provide feedback. In this work, we propose a large language model-based approach to automatically generate test cases and show that they are good measures of student knowledge, using a publicly available dataset that contains student-written Java code. We also discuss future research directions centered on using test cases to help students.
APA
Ashok Kumar, N. & Lan, A.S.. (2024). Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education. Proceedings of the 2024 AAAI Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 257:170-179 Available from https://proceedings.mlr.press/v257/kumar24.html.

Related Material