A Classroom Study of LLM-generated Feedback Intervention in Introductory Programming

Hasnain Heickal, Andrew Lan
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:12-21, 2026.

Abstract

Large language models (LLMs) are increasingly used to provide automated feedback in introductory programming courses, yet empirical evidence from authentic classroom deployments comparing different feedback modalities remains limited. In this work, we present a large-scale classroom study in which AI-generated feedback was deployed through a randomized protocol in an introductory Python programming course. Students received one of three feedback conditions on incorrect submissions: natural language hints, AI-generated failing test cases, or no AI feedback. We release the resulting dataset, \textsc{ProgFeed}, which captures 6{,}693 submissions from 215 consenting students across 17 labs, including feedback conditions, execution-based performance measures, and fine-grained temporal information. Using this data, we analyze learning trajectories, feedback quality, and submission behavior over repeated attempts. We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity. Our results suggest that the form of AI-generated feedback matters, and that evaluating feedback quality—not just its presence—is essential for understanding its pedagogical impact.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-heickal26a, title = {A Classroom Study of LLM-generated Feedback Intervention in Introductory Programming}, author = {Heickal, Hasnain and Lan, Andrew}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {12--21}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/heickal26a/heickal26a.pdf}, url = {https://proceedings.mlr.press/v339/heickal26a.html}, abstract = {Large language models (LLMs) are increasingly used to provide automated feedback in introductory programming courses, yet empirical evidence from authentic classroom deployments comparing different feedback modalities remains limited. In this work, we present a large-scale classroom study in which AI-generated feedback was deployed through a randomized protocol in an introductory Python programming course. Students received one of three feedback conditions on incorrect submissions: natural language hints, AI-generated failing test cases, or no AI feedback. We release the resulting dataset, \textsc{ProgFeed}, which captures 6{,}693 submissions from 215 consenting students across 17 labs, including feedback conditions, execution-based performance measures, and fine-grained temporal information. Using this data, we analyze learning trajectories, feedback quality, and submission behavior over repeated attempts. We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity. Our results suggest that the form of AI-generated feedback matters, and that evaluating feedback quality—not just its presence—is essential for understanding its pedagogical impact.} }
Endnote
%0 Conference Paper %T A Classroom Study of LLM-generated Feedback Intervention in Introductory Programming %A Hasnain Heickal %A Andrew Lan %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-heickal26a %I PMLR %P 12--21 %U https://proceedings.mlr.press/v339/heickal26a.html %V 339 %X Large language models (LLMs) are increasingly used to provide automated feedback in introductory programming courses, yet empirical evidence from authentic classroom deployments comparing different feedback modalities remains limited. In this work, we present a large-scale classroom study in which AI-generated feedback was deployed through a randomized protocol in an introductory Python programming course. Students received one of three feedback conditions on incorrect submissions: natural language hints, AI-generated failing test cases, or no AI feedback. We release the resulting dataset, \textsc{ProgFeed}, which captures 6{,}693 submissions from 215 consenting students across 17 labs, including feedback conditions, execution-based performance measures, and fine-grained temporal information. Using this data, we analyze learning trajectories, feedback quality, and submission behavior over repeated attempts. We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity. Our results suggest that the form of AI-generated feedback matters, and that evaluating feedback quality—not just its presence—is essential for understanding its pedagogical impact.
APA
Heickal, H. & Lan, A.. (2026). A Classroom Study of LLM-generated Feedback Intervention in Introductory Programming. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:12-21 Available from https://proceedings.mlr.press/v339/heickal26a.html.

Related Material