Learning to Compare Hints: Combining Insights from Student Logs and Large Language Models

Ted Zhang, Harshith Arun Kumar, Robin Schmucker, Amos Azaria, Tom Mitchell
Proceedings of the 2024 AAAI Conference on Artificial Intelligence, PMLR 257:162-169, 2024.

Abstract

We explore the general problem of learning to predict which teaching actions will result in the best learning outcomes for students in online courses. More specifically, we consider the problem of predicting which hint will most help a student who answers a practice question incorrectly, and who is about to make a second attempt to answer that question. In previous work we showed that log data from thousands of previous students could be used to learn empirically which of several pre-defined hints produces the best learning outcome. However, while that study utilized data from thousands of students submitting millions of responses, it did not consider the actual text of the question, the hint, or the answer. In this paper, we ask the follow-on question “Can we train a machine learned model to examine the text of the question, the answer, and the text of hints, to predict which hint will lead to better learning outcomes?” Our experimental results show that the answer is yes. This is important because the trained model can now be applied to new questions and hints covering related subject matter, to estimate which of the new hints will be most useful, even before testing it on students. Finally, we show that the pairs of hints for which the model makes most accurate predictions are the hint pairs where choosing the right hint has the biggest payoff (i.e., hint pairs for which the difference in learning outcomes is greatest).

Cite this Paper

BibTeX
@InProceedings{pmlr-v257-zhang24a, title = {Learning to Compare Hints: Combining Insights from Student Logs and Large Language Models}, author = {Zhang, Ted and Kumar, Harshith Arun and Schmucker, Robin and Azaria, Amos and Mitchell, Tom}, booktitle = {Proceedings of the 2024 AAAI Conference on Artificial Intelligence}, pages = {162--169}, year = {2024}, editor = {Ananda, Muktha and Malick, Debshila Basu and Burstein, Jill and Liu, Lydia T. and Liu, Zitao and Sharpnack, James and Wang, Zichao and Wang, Serena}, volume = {257}, series = {Proceedings of Machine Learning Research}, month = {26--27 Feb}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v257/main/assets/zhang24a/zhang24a.pdf}, url = {https://proceedings.mlr.press/v257/zhang24a.html}, abstract = {We explore the general problem of learning to predict which teaching actions will result in the best learning outcomes for students in online courses. More specifically, we consider the problem of predicting which hint will most help a student who answers a practice question incorrectly, and who is about to make a second attempt to answer that question. In previous work we showed that log data from thousands of previous students could be used to learn empirically which of several pre-defined hints produces the best learning outcome. However, while that study utilized data from thousands of students submitting millions of responses, it did not consider the actual text of the question, the hint, or the answer. In this paper, we ask the follow-on question “Can we train a machine learned model to examine the text of the question, the answer, and the text of hints, to predict which hint will lead to better learning outcomes?” Our experimental results show that the answer is yes. This is important because the trained model can now be applied to new questions and hints covering related subject matter, to estimate which of the new hints will be most useful, even before testing it on students. Finally, we show that the pairs of hints for which the model makes most accurate predictions are the hint pairs where choosing the right hint has the biggest payoff (i.e., hint pairs for which the difference in learning outcomes is greatest). } }
Endnote
%0 Conference Paper %T Learning to Compare Hints: Combining Insights from Student Logs and Large Language Models %A Ted Zhang %A Harshith Arun Kumar %A Robin Schmucker %A Amos Azaria %A Tom Mitchell %B Proceedings of the 2024 AAAI Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Muktha Ananda %E Debshila Basu Malick %E Jill Burstein %E Lydia T. Liu %E Zitao Liu %E James Sharpnack %E Zichao Wang %E Serena Wang %F pmlr-v257-zhang24a %I PMLR %P 162--169 %U https://proceedings.mlr.press/v257/zhang24a.html %V 257 %X We explore the general problem of learning to predict which teaching actions will result in the best learning outcomes for students in online courses. More specifically, we consider the problem of predicting which hint will most help a student who answers a practice question incorrectly, and who is about to make a second attempt to answer that question. In previous work we showed that log data from thousands of previous students could be used to learn empirically which of several pre-defined hints produces the best learning outcome. However, while that study utilized data from thousands of students submitting millions of responses, it did not consider the actual text of the question, the hint, or the answer. In this paper, we ask the follow-on question “Can we train a machine learned model to examine the text of the question, the answer, and the text of hints, to predict which hint will lead to better learning outcomes?” Our experimental results show that the answer is yes. This is important because the trained model can now be applied to new questions and hints covering related subject matter, to estimate which of the new hints will be most useful, even before testing it on students. Finally, we show that the pairs of hints for which the model makes most accurate predictions are the hint pairs where choosing the right hint has the biggest payoff (i.e., hint pairs for which the difference in learning outcomes is greatest).
APA
Zhang, T., Kumar, H.A., Schmucker, R., Azaria, A. & Mitchell, T.. (2024). Learning to Compare Hints: Combining Insights from Student Logs and Large Language Models. Proceedings of the 2024 AAAI Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 257:162-169 Available from https://proceedings.mlr.press/v257/zhang24a.html.

Related Material