Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V Chawla
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:70246-70269, 2026.

Abstract

Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS’s ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability preserves the global performance structure. At the same time, provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23–31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks available at https://github.com/Peiyu-Georgia-Li/ATLAS.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-li26fx, title = {Adaptive Testing for {LLM} Evaluation: A Psychometric Alternative to Static Benchmarks}, author = {Li, Peiyu and Tang, Xiuxiu and Chen, Si and Cheng, Ying and Metoyer, Ronald and Hua, Ting and Chawla, Nitesh V}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {70246--70269}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26fx/li26fx.pdf}, url = {https://proceedings.mlr.press/v306/li26fx.html}, abstract = {Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS’s ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability preserves the global performance structure. At the same time, provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23–31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks available at https://github.com/Peiyu-Georgia-Li/ATLAS.} }
Endnote
%0 Conference Paper %T Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks %A Peiyu Li %A Xiuxiu Tang %A Si Chen %A Ying Cheng %A Ronald Metoyer %A Ting Hua %A Nitesh V Chawla %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26fx %I PMLR %P 70246--70269 %U https://proceedings.mlr.press/v306/li26fx.html %V 306 %X Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS’s ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability preserves the global performance structure. At the same time, provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23–31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks available at https://github.com/Peiyu-Georgia-Li/ATLAS.
APA
Li, P., Tang, X., Chen, S., Cheng, Y., Metoyer, R., Hua, T. & Chawla, N.V.. (2026). Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:70246-70269 Available from https://proceedings.mlr.press/v306/li26fx.html.

Related Material