Beyond Aggregate Accuracy: Continuous Evaluation of Residual Failures in Image-Required Educational Mathematics

Ethan Croteau, Neil Heffernan
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:170-179, 2026.

Abstract

Rapid model progress can make educational benchmark results stale, but responsible evaluation requires more than refreshing aggregate scores. We revisit a benchmark of 376 curriculum-authentic, image-required middle-school mathematics items using GPT-5.5 and compare its behavior with representative prior models. In the \textit{with-images} condition, accuracy increases from about 88% for GPT-5.4 to about 93% for GPT-5.5, while the strict never-solved set decreases from 31 items to 17. In contrast, \textit{without-images} performance remains near zero, confirming that the benchmark remains genuinely image-required. We then audit this residual set and find that its composition clarifies what the remaining failures mean: they include inaccessible-image cases, insufficient-context items, answer-format mismatches, and benchmark-quality issues rather than only generic visual-reasoning failures. We treat representation-sensitive probes as future work, using the audit to identify which residual items are appropriate probe candidates. We argue that continuous evaluation of educational AI systems should pair aggregate performance updates with residual item audits that distinguish model limitations from dataset, scoring, accessibility, and task-specification issues.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-croteau26a, title = {Beyond Aggregate Accuracy: Continuous Evaluation of Residual Failures in Image-Required Educational Mathematics}, author = {Croteau, Ethan and Heffernan, Neil}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {170--179}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/croteau26a/croteau26a.pdf}, url = {https://proceedings.mlr.press/v339/croteau26a.html}, abstract = {Rapid model progress can make educational benchmark results stale, but responsible evaluation requires more than refreshing aggregate scores. We revisit a benchmark of 376 curriculum-authentic, image-required middle-school mathematics items using GPT-5.5 and compare its behavior with representative prior models. In the \textit{with-images} condition, accuracy increases from about 88% for GPT-5.4 to about 93% for GPT-5.5, while the strict never-solved set decreases from 31 items to 17. In contrast, \textit{without-images} performance remains near zero, confirming that the benchmark remains genuinely image-required. We then audit this residual set and find that its composition clarifies what the remaining failures mean: they include inaccessible-image cases, insufficient-context items, answer-format mismatches, and benchmark-quality issues rather than only generic visual-reasoning failures. We treat representation-sensitive probes as future work, using the audit to identify which residual items are appropriate probe candidates. We argue that continuous evaluation of educational AI systems should pair aggregate performance updates with residual item audits that distinguish model limitations from dataset, scoring, accessibility, and task-specification issues.} }
Endnote
%0 Conference Paper %T Beyond Aggregate Accuracy: Continuous Evaluation of Residual Failures in Image-Required Educational Mathematics %A Ethan Croteau %A Neil Heffernan %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-croteau26a %I PMLR %P 170--179 %U https://proceedings.mlr.press/v339/croteau26a.html %V 339 %X Rapid model progress can make educational benchmark results stale, but responsible evaluation requires more than refreshing aggregate scores. We revisit a benchmark of 376 curriculum-authentic, image-required middle-school mathematics items using GPT-5.5 and compare its behavior with representative prior models. In the \textit{with-images} condition, accuracy increases from about 88% for GPT-5.4 to about 93% for GPT-5.5, while the strict never-solved set decreases from 31 items to 17. In contrast, \textit{without-images} performance remains near zero, confirming that the benchmark remains genuinely image-required. We then audit this residual set and find that its composition clarifies what the remaining failures mean: they include inaccessible-image cases, insufficient-context items, answer-format mismatches, and benchmark-quality issues rather than only generic visual-reasoning failures. We treat representation-sensitive probes as future work, using the audit to identify which residual items are appropriate probe candidates. We argue that continuous evaluation of educational AI systems should pair aggregate performance updates with residual item audits that distinguish model limitations from dataset, scoring, accessibility, and task-specification issues.
APA
Croteau, E. & Heffernan, N.. (2026). Beyond Aggregate Accuracy: Continuous Evaluation of Residual Failures in Image-Required Educational Mathematics. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:170-179 Available from https://proceedings.mlr.press/v339/croteau26a.html.

Related Material