R.I.P.: Better Models by Survival of the Fittest Prompts

Ping Yu, Weizhe Yuan, Olga Golovneva, Tianhao Wu, Sainbayar Sukhbaatar, Jason E Weston, Jing Xu
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:73350-73374, 2025.

Abstract

Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, from 18th place to 6th overall in the leaderboard.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-yu25u, title = {{R}.{I}.{P}.: Better Models by Survival of the Fittest Prompts}, author = {Yu, Ping and Yuan, Weizhe and Golovneva, Olga and Wu, Tianhao and Sukhbaatar, Sainbayar and Weston, Jason E and Xu, Jing}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {73350--73374}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/yu25u/yu25u.pdf}, url = {https://proceedings.mlr.press/v267/yu25u.html}, abstract = {Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, from 18th place to 6th overall in the leaderboard.} }
Endnote
%0 Conference Paper %T R.I.P.: Better Models by Survival of the Fittest Prompts %A Ping Yu %A Weizhe Yuan %A Olga Golovneva %A Tianhao Wu %A Sainbayar Sukhbaatar %A Jason E Weston %A Jing Xu %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-yu25u %I PMLR %P 73350--73374 %U https://proceedings.mlr.press/v267/yu25u.html %V 267 %X Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, from 18th place to 6th overall in the leaderboard.
APA
Yu, P., Yuan, W., Golovneva, O., Wu, T., Sukhbaatar, S., Weston, J.E. & Xu, J.. (2025). R.I.P.: Better Models by Survival of the Fittest Prompts. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:73350-73374 Available from https://proceedings.mlr.press/v267/yu25u.html.

Related Material