More Sources, Better AI? When Context Helps and Harms AI-Assisted Discovery in Educational Data Archives

Xin Wei, Morgan Lee, Jie Min, Jeremy Roschelle
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:112-122, 2026.

Abstract

Can AI support the research reasoning required to use unfamiliar educational datasets? We evaluated an AI-assisted discovery tool across two educational archives, ASSISTments and SEDA, using 50-item expert-designed assessments. We varied the sources available to the tool: dataset documentation, published papers, synthetic data samples, and their combination. Full triangulation across all three sources outperformed documentation alone in overall accuracy: 89% versus 67% on ASSISTments and 90% versus 73% on SEDA. However, more context was not always better. With codebook-oriented documentation, adding synthetic data without papers reduced appropriate-use accuracy from 80% to 40%, while adding papers without data reduced code-generation accuracy from 90% to 75%. Effective AI-assisted research support requires matching the information sources to how well a dataset’s design fits the research questions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-wei26a, title = {More Sources, Better AI? When Context Helps and Harms AI-Assisted Discovery in Educational Data Archives}, author = {Wei, Xin and Lee, Morgan and Min, Jie and Roschelle, Jeremy}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {112--122}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/wei26a/wei26a.pdf}, url = {https://proceedings.mlr.press/v339/wei26a.html}, abstract = {Can AI support the research reasoning required to use unfamiliar educational datasets? We evaluated an AI-assisted discovery tool across two educational archives, ASSISTments and SEDA, using 50-item expert-designed assessments. We varied the sources available to the tool: dataset documentation, published papers, synthetic data samples, and their combination. Full triangulation across all three sources outperformed documentation alone in overall accuracy: 89% versus 67% on ASSISTments and 90% versus 73% on SEDA. However, more context was not always better. With codebook-oriented documentation, adding synthetic data without papers reduced appropriate-use accuracy from 80% to 40%, while adding papers without data reduced code-generation accuracy from 90% to 75%. Effective AI-assisted research support requires matching the information sources to how well a dataset’s design fits the research questions.} }
Endnote
%0 Conference Paper %T More Sources, Better AI? When Context Helps and Harms AI-Assisted Discovery in Educational Data Archives %A Xin Wei %A Morgan Lee %A Jie Min %A Jeremy Roschelle %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-wei26a %I PMLR %P 112--122 %U https://proceedings.mlr.press/v339/wei26a.html %V 339 %X Can AI support the research reasoning required to use unfamiliar educational datasets? We evaluated an AI-assisted discovery tool across two educational archives, ASSISTments and SEDA, using 50-item expert-designed assessments. We varied the sources available to the tool: dataset documentation, published papers, synthetic data samples, and their combination. Full triangulation across all three sources outperformed documentation alone in overall accuracy: 89% versus 67% on ASSISTments and 90% versus 73% on SEDA. However, more context was not always better. With codebook-oriented documentation, adding synthetic data without papers reduced appropriate-use accuracy from 80% to 40%, while adding papers without data reduced code-generation accuracy from 90% to 75%. Effective AI-assisted research support requires matching the information sources to how well a dataset’s design fits the research questions.
APA
Wei, X., Lee, M., Min, J. & Roschelle, J.. (2026). More Sources, Better AI? When Context Helps and Harms AI-Assisted Discovery in Educational Data Archives. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:112-122 Available from https://proceedings.mlr.press/v339/wei26a.html.

Related Material