OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang, Junhao Zheng, Kexin Yang, Xingzhang Ren, Dayiheng Liu, Linfeng Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:129321-129362, 2026.

Abstract

As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall—LLM pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. It also outperforms previous data selection methods across different stages of training, including from-scratch pre-training and also mid-training. Beyond online selection, the OPUS utility score also demonstrates potential as a static filter for flagging and removing toxic documents from contaminated training corpora prior to training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26gc, title = {{OPUS}: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration}, author = {Wang, Shaobo and Ouyang, Xuan and Xu, Tianyi and Hu, Yuzheng and Liu, Jialin and Chen, Guo and Zhang, Tianyu and Zheng, Junhao and Yang, Kexin and Ren, Xingzhang and Liu, Dayiheng and Zhang, Linfeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {129321--129362}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26gc/wang26gc.pdf}, url = {https://proceedings.mlr.press/v306/wang26gc.html}, abstract = {As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall—LLM pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. It also outperforms previous data selection methods across different stages of training, including from-scratch pre-training and also mid-training. Beyond online selection, the OPUS utility score also demonstrates potential as a static filter for flagging and removing toxic documents from contaminated training corpora prior to training.} }
Endnote
%0 Conference Paper %T OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration %A Shaobo Wang %A Xuan Ouyang %A Tianyi Xu %A Yuzheng Hu %A Jialin Liu %A Guo Chen %A Tianyu Zhang %A Junhao Zheng %A Kexin Yang %A Xingzhang Ren %A Dayiheng Liu %A Linfeng Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26gc %I PMLR %P 129321--129362 %U https://proceedings.mlr.press/v306/wang26gc.html %V 306 %X As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall—LLM pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. It also outperforms previous data selection methods across different stages of training, including from-scratch pre-training and also mid-training. Beyond online selection, the OPUS utility score also demonstrates potential as a static filter for flagging and removing toxic documents from contaminated training corpora prior to training.
APA
Wang, S., Ouyang, X., Xu, T., Hu, Y., Liu, J., Chen, G., Zhang, T., Zheng, J., Yang, K., Ren, X., Liu, D. & Zhang, L.. (2026). OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:129321-129362 Available from https://proceedings.mlr.press/v306/wang26gc.html.

Related Material