CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction

Yinghao Ma, Haiwen Xia, Hewei Gao, Weixiong Chen, Ye Yuxin, Yuchen Yang, Sungkyun Chang, Mingshuo Ding, Yizhi Li, Ruibin Yuan, Simon Dixon, Emmanouil Benetos
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:85099-85130, 2026.

Abstract

While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind, remaining fragmented and narrowly focused. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for Compositional Music Instruction (CMI) reward modeling, where the generated music may be conditioned on text descriptions, lyrics, and/or audio prompts. We first introduce CMIPref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMIRewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text–music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgments scores on Music Arena and CMI-Pref test set, as well as preference agreement on Music Arena and CMI-Pref. Additional analyses examine performance variation across factors such as annotators, annotation timing and confidence, music generation models, and audio length. Experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via topk filtering. Our work provides the necessary data, benchmarks, and models to advance aligned music generation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ma26am, title = {{CMI}-{R}eward{B}ench: Evaluating Music Reward Models with Compositional Multimodal Instruction}, author = {Ma, Yinghao and Xia, Haiwen and Gao, Hewei and Chen, Weixiong and Yuxin, Ye and Yang, Yuchen and Chang, Sungkyun and Ding, Mingshuo and Li, Yizhi and Yuan, Ruibin and Dixon, Simon and Benetos, Emmanouil}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {85099--85130}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ma26am/ma26am.pdf}, url = {https://proceedings.mlr.press/v306/ma26am.html}, abstract = {While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind, remaining fragmented and narrowly focused. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for Compositional Music Instruction (CMI) reward modeling, where the generated music may be conditioned on text descriptions, lyrics, and/or audio prompts. We first introduce CMIPref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMIRewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text–music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgments scores on Music Arena and CMI-Pref test set, as well as preference agreement on Music Arena and CMI-Pref. Additional analyses examine performance variation across factors such as annotators, annotation timing and confidence, music generation models, and audio length. Experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via topk filtering. Our work provides the necessary data, benchmarks, and models to advance aligned music generation.} }
Endnote
%0 Conference Paper %T CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction %A Yinghao Ma %A Haiwen Xia %A Hewei Gao %A Weixiong Chen %A Ye Yuxin %A Yuchen Yang %A Sungkyun Chang %A Mingshuo Ding %A Yizhi Li %A Ruibin Yuan %A Simon Dixon %A Emmanouil Benetos %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ma26am %I PMLR %P 85099--85130 %U https://proceedings.mlr.press/v306/ma26am.html %V 306 %X While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind, remaining fragmented and narrowly focused. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for Compositional Music Instruction (CMI) reward modeling, where the generated music may be conditioned on text descriptions, lyrics, and/or audio prompts. We first introduce CMIPref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMIRewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text–music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgments scores on Music Arena and CMI-Pref test set, as well as preference agreement on Music Arena and CMI-Pref. Additional analyses examine performance variation across factors such as annotators, annotation timing and confidence, music generation models, and audio length. Experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via topk filtering. Our work provides the necessary data, benchmarks, and models to advance aligned music generation.
APA
Ma, Y., Xia, H., Gao, H., Chen, W., Yuxin, Y., Yang, Y., Chang, S., Ding, M., Li, Y., Yuan, R., Dixon, S. & Benetos, E.. (2026). CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:85099-85130 Available from https://proceedings.mlr.press/v306/ma26am.html.

Related Material