Leaderboard Incentives: Model Rankings under Strategic Post-Training

Yatong Chen, Guanhua Zhang, Moritz Hardt
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18142-18165, 2026.

Abstract

Influential benchmarks incentivize competing model builders to strategically allocate post-training resources towards improvements on the leaderboard, a phenomenon dubbed benchmaxxing or training on the test task. In this work, we initiate a principled study of the incentive structure that benchmarks induce. We model benchmarking as a Stackelberg game between a benchmark designer who chooses an evaluation protocol and multiple model providers who compete simultaneously in a subgame given by the designer’s choice. Each competitor has a model of unknown latent quality and can inflate its observed score by allocating resources to benchmark-specific improvements. First, we prove that current benchmarks induce games for which no Nash equilibrium between model developers exists. This result suggests one explanation for why current practice leads to misaligned incentives, prompting model providers to strategize in opaque ways. However, we prove that under mild conditions, a recently proposed evaluation protocol, called tune-before-test, induces a benchmark with a unique Nash equilibrium that ranks models by latent quality. This positive result demonstrates that benchmarks need not set bad incentives, even if current evaluations do.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26gf, title = {Leaderboard Incentives: Model Rankings under Strategic Post-Training}, author = {Chen, Yatong and Zhang, Guanhua and Hardt, Moritz}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18142--18165}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26gf/chen26gf.pdf}, url = {https://proceedings.mlr.press/v306/chen26gf.html}, abstract = {Influential benchmarks incentivize competing model builders to strategically allocate post-training resources towards improvements on the leaderboard, a phenomenon dubbed benchmaxxing or training on the test task. In this work, we initiate a principled study of the incentive structure that benchmarks induce. We model benchmarking as a Stackelberg game between a benchmark designer who chooses an evaluation protocol and multiple model providers who compete simultaneously in a subgame given by the designer’s choice. Each competitor has a model of unknown latent quality and can inflate its observed score by allocating resources to benchmark-specific improvements. First, we prove that current benchmarks induce games for which no Nash equilibrium between model developers exists. This result suggests one explanation for why current practice leads to misaligned incentives, prompting model providers to strategize in opaque ways. However, we prove that under mild conditions, a recently proposed evaluation protocol, called tune-before-test, induces a benchmark with a unique Nash equilibrium that ranks models by latent quality. This positive result demonstrates that benchmarks need not set bad incentives, even if current evaluations do.} }
Endnote
%0 Conference Paper %T Leaderboard Incentives: Model Rankings under Strategic Post-Training %A Yatong Chen %A Guanhua Zhang %A Moritz Hardt %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26gf %I PMLR %P 18142--18165 %U https://proceedings.mlr.press/v306/chen26gf.html %V 306 %X Influential benchmarks incentivize competing model builders to strategically allocate post-training resources towards improvements on the leaderboard, a phenomenon dubbed benchmaxxing or training on the test task. In this work, we initiate a principled study of the incentive structure that benchmarks induce. We model benchmarking as a Stackelberg game between a benchmark designer who chooses an evaluation protocol and multiple model providers who compete simultaneously in a subgame given by the designer’s choice. Each competitor has a model of unknown latent quality and can inflate its observed score by allocating resources to benchmark-specific improvements. First, we prove that current benchmarks induce games for which no Nash equilibrium between model developers exists. This result suggests one explanation for why current practice leads to misaligned incentives, prompting model providers to strategize in opaque ways. However, we prove that under mild conditions, a recently proposed evaluation protocol, called tune-before-test, induces a benchmark with a unique Nash equilibrium that ranks models by latent quality. This positive result demonstrates that benchmarks need not set bad incentives, even if current evaluations do.
APA
Chen, Y., Zhang, G. & Hardt, M.. (2026). Leaderboard Incentives: Model Rankings under Strategic Post-Training. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18142-18165 Available from https://proceedings.mlr.press/v306/chen26gf.html.

Related Material