AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations

Kasra Lekan, Zachary A. Pardos
Proceedings of the 2024 AAAI Conference on Artificial Intelligence, PMLR 257:85-96, 2024.

Abstract

Choosing an undergraduate major is an important decision that impacts academic and career outcomes. We investigate using GPT-4, a state-of-the-art large language model (LLM), to augment human advising for major selection. Through a 3-phase survey, we compare GPT suggestions and responses for undeclared Freshmen and Sophomore students (n=33) to expert responses from university advisors (n=25). Undeclared students were first surveyed on their interests and goals. These responses were then given to both campus advisors and to GPT to produce a major recommendation for each student. In the case of GPT, information about the majors offered on campus was added to the prompt. Advisors, overall, rated the recommendations of GPT to be highly helpful and agreed with their recommendations 33% of the time. Additionally, we observe more agreement with AI major recommendations when advisors see the AI recommendations before making their own. However, this result was not statistically significant. The results provide a first signal as to the viability of LLMs for personalized major recommendation and shed light on the promise and limitations of AI for advising support.

Cite this Paper

BibTeX
@InProceedings{pmlr-v257-lekan24a, title = {AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations}, author = {Lekan, Kasra and Pardos, Zachary A.}, booktitle = {Proceedings of the 2024 AAAI Conference on Artificial Intelligence}, pages = {85--96}, year = {2024}, editor = {Ananda, Muktha and Malick, Debshila Basu and Burstein, Jill and Liu, Lydia T. and Liu, Zitao and Sharpnack, James and Wang, Zichao and Wang, Serena}, volume = {257}, series = {Proceedings of Machine Learning Research}, month = {26--27 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v257/main/assets/lekan24a/lekan24a.pdf}, url = {https://proceedings.mlr.press/v257/lekan24a.html}, abstract = {Choosing an undergraduate major is an important decision that impacts academic and career outcomes. We investigate using GPT-4, a state-of-the-art large language model (LLM), to augment human advising for major selection. Through a 3-phase survey, we compare GPT suggestions and responses for undeclared Freshmen and Sophomore students (n=33) to expert responses from university advisors (n=25). Undeclared students were first surveyed on their interests and goals. These responses were then given to both campus advisors and to GPT to produce a major recommendation for each student. In the case of GPT, information about the majors offered on campus was added to the prompt. Advisors, overall, rated the recommendations of GPT to be highly helpful and agreed with their recommendations 33% of the time. Additionally, we observe more agreement with AI major recommendations when advisors see the AI recommendations before making their own. However, this result was not statistically significant. The results provide a first signal as to the viability of LLMs for personalized major recommendation and shed light on the promise and limitations of AI for advising support.} }
Endnote
%0 Conference Paper %T AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations %A Kasra Lekan %A Zachary A. Pardos %B Proceedings of the 2024 AAAI Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Muktha Ananda %E Debshila Basu Malick %E Jill Burstein %E Lydia T. Liu %E Zitao Liu %E James Sharpnack %E Zichao Wang %E Serena Wang %F pmlr-v257-lekan24a %I PMLR %P 85--96 %U https://proceedings.mlr.press/v257/lekan24a.html %V 257 %X Choosing an undergraduate major is an important decision that impacts academic and career outcomes. We investigate using GPT-4, a state-of-the-art large language model (LLM), to augment human advising for major selection. Through a 3-phase survey, we compare GPT suggestions and responses for undeclared Freshmen and Sophomore students (n=33) to expert responses from university advisors (n=25). Undeclared students were first surveyed on their interests and goals. These responses were then given to both campus advisors and to GPT to produce a major recommendation for each student. In the case of GPT, information about the majors offered on campus was added to the prompt. Advisors, overall, rated the recommendations of GPT to be highly helpful and agreed with their recommendations 33% of the time. Additionally, we observe more agreement with AI major recommendations when advisors see the AI recommendations before making their own. However, this result was not statistically significant. The results provide a first signal as to the viability of LLMs for personalized major recommendation and shed light on the promise and limitations of AI for advising support.
APA
Lekan, K. & Pardos, Z.A.. (2024). AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations. Proceedings of the 2024 AAAI Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 257:85-96 Available from https://proceedings.mlr.press/v257/lekan24a.html.

Related Material