Learning More from Less: Unlocking Internal Representations for Benchmark Compression

Yueqi Zhang, Jin Hu, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yiwei Li, Jiayi Shi, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:155744-155784, 2026.

Abstract

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model’s decision process and fail to capture information encoded in hidden states. To address this, we introduce RepCore, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, RepCore achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26bj, title = {Learning More from Less: Unlocking Internal Representations for Benchmark Compression}, author = {Zhang, Yueqi and Hu, Jin and Feng, Shaoxiong and Yuan, Peiwen and Wang, Xinglin and Li, Yiwei and Shi, Jiayi and Tan, Chuyi and Zhang, Ji and Pan, Boyuan and Hu, Yao and Li, Kan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {155744--155784}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26bj/zhang26bj.pdf}, url = {https://proceedings.mlr.press/v306/zhang26bj.html}, abstract = {The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model’s decision process and fail to capture information encoded in hidden states. To address this, we introduce RepCore, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, RepCore achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.} }
Endnote
%0 Conference Paper %T Learning More from Less: Unlocking Internal Representations for Benchmark Compression %A Yueqi Zhang %A Jin Hu %A Shaoxiong Feng %A Peiwen Yuan %A Xinglin Wang %A Yiwei Li %A Jiayi Shi %A Chuyi Tan %A Ji Zhang %A Boyuan Pan %A Yao Hu %A Kan Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26bj %I PMLR %P 155744--155784 %U https://proceedings.mlr.press/v306/zhang26bj.html %V 306 %X The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model’s decision process and fail to capture information encoded in hidden states. To address this, we introduce RepCore, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, RepCore achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.
APA
Zhang, Y., Hu, J., Feng, S., Yuan, P., Wang, X., Li, Y., Shi, J., Tan, C., Zhang, J., Pan, B., Hu, Y. & Li, K.. (2026). Learning More from Less: Unlocking Internal Representations for Benchmark Compression. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:155744-155784 Available from https://proceedings.mlr.press/v306/zhang26bj.html.

Related Material