AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals

Gati Aher, Robin Schmucker, Tom Mitchell, Zachary C. Lipton
Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop, PMLR 273:1-40, 2025.

Abstract

When executed well, project-based learning (PBL) engages students’ intrinsic motivation, encourages students to learn far beyond a course’s limited curriculum, and prepares students to think critically and maturely about the skills and tools at their disposal. However, educators experience mixed results when using PBL in their classrooms: some students thrive with minimal guidance and others flounder. Early evaluation of project proposals could help educators determine which students need more support, yet evaluating project proposals and student aptitude is time-consuming and difficult to scale. In this work, we design, implement, and conduct an initial user study (n= 36) for a software system that collects project proposals and aptitude information to support educators in determining whether a student is ready to engage with PBL. We find that (1) users perceived the system as helpful for writing project proposals and identifying tools and technologies to learn more about, (2) educator ratings indicate that users with less technical experience in the project topic tend to write lower-quality project proposals, and (3) GPT-4o’s ratings show agreement with educator ratings. While the prospect of using LLMs to rate the quality of students’ project proposals is promising, its long-term effectiveness strongly hinges on future efforts at characterizing indicators that reliably predict students’ success and motivation to learn.

Cite this Paper

BibTeX
@InProceedings{pmlr-v273-aher25a, title = {AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals}, author = {Aher, Gati and Schmucker, Robin and Mitchell, Tom and Lipton, Zachary C.}, booktitle = {Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop}, pages = {1--40}, year = {2025}, editor = {Wang, Zichao and Woodhead, Simon and Ananda, Muktha and Mallick, Debshila Basu and Sharpnack, James and Burstein, Jill}, volume = {273}, series = {Proceedings of Machine Learning Research}, month = {03 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v273/main/assets/aher25a/aher25a.pdf}, url = {https://proceedings.mlr.press/v273/aher25a.html}, abstract = {When executed well, project-based learning (PBL) engages students’ intrinsic motivation, encourages students to learn far beyond a course’s limited curriculum, and prepares students to think critically and maturely about the skills and tools at their disposal. However, educators experience mixed results when using PBL in their classrooms: some students thrive with minimal guidance and others flounder. Early evaluation of project proposals could help educators determine which students need more support, yet evaluating project proposals and student aptitude is time-consuming and difficult to scale. In this work, we design, implement, and conduct an initial user study (n= 36) for a software system that collects project proposals and aptitude information to support educators in determining whether a student is ready to engage with PBL. We find that (1) users perceived the system as helpful for writing project proposals and identifying tools and technologies to learn more about, (2) educator ratings indicate that users with less technical experience in the project topic tend to write lower-quality project proposals, and (3) GPT-4o’s ratings show agreement with educator ratings. While the prospect of using LLMs to rate the quality of students’ project proposals is promising, its long-term effectiveness strongly hinges on future efforts at characterizing indicators that reliably predict students’ success and motivation to learn.} }
Endnote
%0 Conference Paper %T AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals %A Gati Aher %A Robin Schmucker %A Tom Mitchell %A Zachary C. Lipton %B Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop %C Proceedings of Machine Learning Research %D 2025 %E Zichao Wang %E Simon Woodhead %E Muktha Ananda %E Debshila Basu Mallick %E James Sharpnack %E Jill Burstein %F pmlr-v273-aher25a %I PMLR %P 1--40 %U https://proceedings.mlr.press/v273/aher25a.html %V 273 %X When executed well, project-based learning (PBL) engages students’ intrinsic motivation, encourages students to learn far beyond a course’s limited curriculum, and prepares students to think critically and maturely about the skills and tools at their disposal. However, educators experience mixed results when using PBL in their classrooms: some students thrive with minimal guidance and others flounder. Early evaluation of project proposals could help educators determine which students need more support, yet evaluating project proposals and student aptitude is time-consuming and difficult to scale. In this work, we design, implement, and conduct an initial user study (n= 36) for a software system that collects project proposals and aptitude information to support educators in determining whether a student is ready to engage with PBL. We find that (1) users perceived the system as helpful for writing project proposals and identifying tools and technologies to learn more about, (2) educator ratings indicate that users with less technical experience in the project topic tend to write lower-quality project proposals, and (3) GPT-4o’s ratings show agreement with educator ratings. While the prospect of using LLMs to rate the quality of students’ project proposals is promising, its long-term effectiveness strongly hinges on future efforts at characterizing indicators that reliably predict students’ success and motivation to learn.
APA
Aher, G., Schmucker, R., Mitchell, T. & Lipton, Z.C.. (2025). AI Mentors for Student Projects: Spotting Early Issues in Computer Science Proposals. Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop, in Proceedings of Machine Learning Research 273:1-40 Available from https://proceedings.mlr.press/v273/aher25a.html.

Related Material