Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

Haochun Wang, Chaofen Yang, Jiatong Liu, Jingbo Wang, Zewen Qiang, Sendong Zhao, Bing Qin, Ting Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:131674-131691, 2026.

Abstract

In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because candidate contexts must be validated with repeated LLM calls. We argue that demonstration selection is easier to judge than to find: predicting whether a specific query–context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate each training query’s success rate, trains a lightweight router to predict difficulty from the query, and trains level-specific judges to score sampled contexts. At inference, DiSP performs stop-on-acceptance judging under an explicit budget and typically makes a single LLM call, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama 3–8B and Qwen 2.5–7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4%, while achieving up to 23$\times$ end-to-end wall-clock speedup.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wang26jp, title = {Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection}, author = {Wang, Haochun and Yang, Chaofen and Liu, Jiatong and Wang, Jingbo and Qiang, Zewen and Zhao, Sendong and Qin, Bing and Liu, Ting}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {131674--131691}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26jp/wang26jp.pdf}, url = {https://proceedings.mlr.press/v306/wang26jp.html}, abstract = {In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because candidate contexts must be validated with repeated LLM calls. We argue that demonstration selection is easier to judge than to find: predicting whether a specific query–context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate each training query’s success rate, trains a lightweight router to predict difficulty from the query, and trains level-specific judges to score sampled contexts. At inference, DiSP performs stop-on-acceptance judging under an explicit budget and typically makes a single LLM call, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama 3–8B and Qwen 2.5–7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4%, while achieving up to 23$\times$ end-to-end wall-clock speedup.} }
Endnote
%0 Conference Paper %T Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection %A Haochun Wang %A Chaofen Yang %A Jiatong Liu %A Jingbo Wang %A Zewen Qiang %A Sendong Zhao %A Bing Qin %A Ting Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26jp %I PMLR %P 131674--131691 %U https://proceedings.mlr.press/v306/wang26jp.html %V 306 %X In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because candidate contexts must be validated with repeated LLM calls. We argue that demonstration selection is easier to judge than to find: predicting whether a specific query–context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate each training query’s success rate, trains a lightweight router to predict difficulty from the query, and trains level-specific judges to score sampled contexts. At inference, DiSP performs stop-on-acceptance judging under an explicit budget and typically makes a single LLM call, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama 3–8B and Qwen 2.5–7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4%, while achieving up to 23$\times$ end-to-end wall-clock speedup.
APA
Wang, H., Yang, C., Liu, J., Wang, J., Qiang, Z., Zhao, S., Qin, B. & Liu, T.. (2026). Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:131674-131691 Available from https://proceedings.mlr.press/v306/wang26jp.html.

Related Material