Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

Zhiqi Yu, Xingping Liu, Haobin Mao, Mingshuo Liu, Long Chen, Jack Xin, Yifeng Yu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:150670-150695, 2026.

Abstract

Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from the University of California, Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of handwritten, in-person proctored free-response quiz submissions from nearly 800 students included in the paper’s empirical analysis. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable real-course deployment. Building on the dataset and evaluation framework developed here, we outline a path toward a future standardized benchmark for AI grading of handwritten mathematics to support reproducible evaluation, transparent comparison, reliable deployment, and future research.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yu26ac, title = {Evaluating {AI} Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark}, author = {Yu, Zhiqi and Liu, Xingping and Mao, Haobin and Liu, Mingshuo and Chen, Long and Xin, Jack and Yu, Yifeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {150670--150695}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yu26ac/yu26ac.pdf}, url = {https://proceedings.mlr.press/v306/yu26ac.html}, abstract = {Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from the University of California, Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of handwritten, in-person proctored free-response quiz submissions from nearly 800 students included in the paper’s empirical analysis. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable real-course deployment. Building on the dataset and evaluation framework developed here, we outline a path toward a future standardized benchmark for AI grading of handwritten mathematics to support reproducible evaluation, transparent comparison, reliable deployment, and future research.} }
Endnote
%0 Conference Paper %T Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark %A Zhiqi Yu %A Xingping Liu %A Haobin Mao %A Mingshuo Liu %A Long Chen %A Jack Xin %A Yifeng Yu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yu26ac %I PMLR %P 150670--150695 %U https://proceedings.mlr.press/v306/yu26ac.html %V 306 %X Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from the University of California, Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of handwritten, in-person proctored free-response quiz submissions from nearly 800 students included in the paper’s empirical analysis. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable real-course deployment. Building on the dataset and evaluation framework developed here, we outline a path toward a future standardized benchmark for AI grading of handwritten mathematics to support reproducible evaluation, transparent comparison, reliable deployment, and future research.
APA
Yu, Z., Liu, X., Mao, H., Liu, M., Chen, L., Xin, J. & Yu, Y.. (2026). Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:150670-150695 Available from https://proceedings.mlr.press/v306/yu26ac.html.

Related Material