Evaluating the Landscape of Automated Scoring with GenAI: A Systematic Review

Warren Li, Melanie Kurimchak, Alexandra Andres, John Whitmer
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:140-145, 2026.

Abstract

As Generative Artificial Intelligence (GenAI) capabilities rapidly expand in educational settings, assessing the validity, reliability, and fairness of these tools is critical before widespread classroom deployment. While there has been a large amount of research, it is not yet clear what methods are being used to evaluate these foundational concepts and what the findings are. We present a work-in-progress systematic literature review focusing exclusively on automated item scoring. Based on a final coded corpus of 107 empirical studies published since January 2023, our preliminary analysis reveals that while prompt-based Large Language Models (LLMs) are achieving parity with traditional transformer-based methods, significant gaps remain regarding fairness evaluation and the representation of diverse student populations. Quadratic weighted kappa (QWK) is the dominant agreement metric, used in 41.8% of studies, and fairness is acknowledged by 55% of studies but empirically evaluated in only 11.2%. This paper offers evidence-based guidance for responsible classroom integration of AI.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-li26a, title = {Evaluating the Landscape of Automated Scoring with GenAI: A Systematic Review}, author = {Li, Warren and Kurimchak, Melanie and Andres, Alexandra and Whitmer, John}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {140--145}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/li26a/li26a.pdf}, url = {https://proceedings.mlr.press/v339/li26a.html}, abstract = {As Generative Artificial Intelligence (GenAI) capabilities rapidly expand in educational settings, assessing the validity, reliability, and fairness of these tools is critical before widespread classroom deployment. While there has been a large amount of research, it is not yet clear what methods are being used to evaluate these foundational concepts and what the findings are. We present a work-in-progress systematic literature review focusing exclusively on automated item scoring. Based on a final coded corpus of 107 empirical studies published since January 2023, our preliminary analysis reveals that while prompt-based Large Language Models (LLMs) are achieving parity with traditional transformer-based methods, significant gaps remain regarding fairness evaluation and the representation of diverse student populations. Quadratic weighted kappa (QWK) is the dominant agreement metric, used in 41.8% of studies, and fairness is acknowledged by 55% of studies but empirically evaluated in only 11.2%. This paper offers evidence-based guidance for responsible classroom integration of AI.} }
Endnote
%0 Conference Paper %T Evaluating the Landscape of Automated Scoring with GenAI: A Systematic Review %A Warren Li %A Melanie Kurimchak %A Alexandra Andres %A John Whitmer %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-li26a %I PMLR %P 140--145 %U https://proceedings.mlr.press/v339/li26a.html %V 339 %X As Generative Artificial Intelligence (GenAI) capabilities rapidly expand in educational settings, assessing the validity, reliability, and fairness of these tools is critical before widespread classroom deployment. While there has been a large amount of research, it is not yet clear what methods are being used to evaluate these foundational concepts and what the findings are. We present a work-in-progress systematic literature review focusing exclusively on automated item scoring. Based on a final coded corpus of 107 empirical studies published since January 2023, our preliminary analysis reveals that while prompt-based Large Language Models (LLMs) are achieving parity with traditional transformer-based methods, significant gaps remain regarding fairness evaluation and the representation of diverse student populations. Quadratic weighted kappa (QWK) is the dominant agreement metric, used in 41.8% of studies, and fairness is acknowledged by 55% of studies but empirically evaluated in only 11.2%. This paper offers evidence-based guidance for responsible classroom integration of AI.
APA
Li, W., Kurimchak, M., Andres, A. & Whitmer, J.. (2026). Evaluating the Landscape of Automated Scoring with GenAI: A Systematic Review. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:140-145 Available from https://proceedings.mlr.press/v339/li26a.html.

Related Material