EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations

Baptiste Moreau-Pernet
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:180-186, 2026.

Abstract

Conversational learner simulations are valuable tools for testing learning theories, evaluating instructional materials and automated tutors, or powering teachable agents. Recently, large language models (LLM) have enabled richer, more naturalistic interactions with simulated learners; however, no open framework exists for evaluating whether such simulations faithfully reproduce real learner behavior. We introduce \textbf{EvalConvoLearn}, an open-source framework that assesses learner simulations along two axes: \emph{learning behavior} (skill-conditioned mastery outcomes) and \emph{conversational quality} (talk moves, error type distributions, question rate, turn length). EvalConvoLearn measures how closely a simulated learner approximates answer distributions observed in data by grounding metrics in authentic tutoring conversation datasets, and anchoring generated tutor responses in existing tutor utterances. The framework is demonstrated on a dataset of tutoring dialogues, including results for two LLM-based learner simulations, and the published GitHub code.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-moreau-pernet26a, title = {EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations}, author = {Moreau-Pernet, Baptiste}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {180--186}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/moreau-pernet26a/moreau-pernet26a.pdf}, url = {https://proceedings.mlr.press/v339/moreau-pernet26a.html}, abstract = {Conversational learner simulations are valuable tools for testing learning theories, evaluating instructional materials and automated tutors, or powering teachable agents. Recently, large language models (LLM) have enabled richer, more naturalistic interactions with simulated learners; however, no open framework exists for evaluating whether such simulations faithfully reproduce real learner behavior. We introduce \textbf{EvalConvoLearn}, an open-source framework that assesses learner simulations along two axes: \emph{learning behavior} (skill-conditioned mastery outcomes) and \emph{conversational quality} (talk moves, error type distributions, question rate, turn length). EvalConvoLearn measures how closely a simulated learner approximates answer distributions observed in data by grounding metrics in authentic tutoring conversation datasets, and anchoring generated tutor responses in existing tutor utterances. The framework is demonstrated on a dataset of tutoring dialogues, including results for two LLM-based learner simulations, and the published GitHub code.} }
Endnote
%0 Conference Paper %T EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations %A Baptiste Moreau-Pernet %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-moreau-pernet26a %I PMLR %P 180--186 %U https://proceedings.mlr.press/v339/moreau-pernet26a.html %V 339 %X Conversational learner simulations are valuable tools for testing learning theories, evaluating instructional materials and automated tutors, or powering teachable agents. Recently, large language models (LLM) have enabled richer, more naturalistic interactions with simulated learners; however, no open framework exists for evaluating whether such simulations faithfully reproduce real learner behavior. We introduce \textbf{EvalConvoLearn}, an open-source framework that assesses learner simulations along two axes: \emph{learning behavior} (skill-conditioned mastery outcomes) and \emph{conversational quality} (talk moves, error type distributions, question rate, turn length). EvalConvoLearn measures how closely a simulated learner approximates answer distributions observed in data by grounding metrics in authentic tutoring conversation datasets, and anchoring generated tutor responses in existing tutor utterances. The framework is demonstrated on a dataset of tutoring dialogues, including results for two LLM-based learner simulations, and the published GitHub code.
APA
Moreau-Pernet, B.. (2026). EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:180-186 Available from https://proceedings.mlr.press/v339/moreau-pernet26a.html.

Related Material