TMD-Bench: A Multi-Level Evaluation Paradigm for Music–Dance Co-Generation

Xiaoda Yang, Majun Zhang, Changhao Pan, Nick Huang, Yang Yuguang, Fan Zhuo, Pengfei Zhou, Jin Zhou, Sizhe Shan, Shan Yang, Miles Yang, Yang You, Zhou Zhao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:147062-147084, 2026.

Abstract

Unified audio–visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio–video synthesis to music–dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music–dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music–dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio–visual models (e.g., Veo 3, Sora 2) produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music–dance models that explicitly optimize rhythmic and kinetic coherence. The code is available at https://github.com/MM-Speech/TMD-Bench.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yang26dl, title = {{TMD}-Bench: A Multi-Level Evaluation Paradigm for Music–Dance Co-Generation}, author = {Yang, Xiaoda and Zhang, Majun and Pan, Changhao and Huang, Nick and Yuguang, Yang and Zhuo, Fan and Zhou, Pengfei and Zhou, Jin and Shan, Sizhe and Yang, Shan and Yang, Miles and You, Yang and Zhao, Zhou}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {147062--147084}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yang26dl/yang26dl.pdf}, url = {https://proceedings.mlr.press/v306/yang26dl.html}, abstract = {Unified audio–visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio–video synthesis to music–dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music–dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music–dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio–visual models (e.g., Veo 3, Sora 2) produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music–dance models that explicitly optimize rhythmic and kinetic coherence. The code is available at https://github.com/MM-Speech/TMD-Bench.} }
Endnote
%0 Conference Paper %T TMD-Bench: A Multi-Level Evaluation Paradigm for Music–Dance Co-Generation %A Xiaoda Yang %A Majun Zhang %A Changhao Pan %A Nick Huang %A Yang Yuguang %A Fan Zhuo %A Pengfei Zhou %A Jin Zhou %A Sizhe Shan %A Shan Yang %A Miles Yang %A Yang You %A Zhou Zhao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yang26dl %I PMLR %P 147062--147084 %U https://proceedings.mlr.press/v306/yang26dl.html %V 306 %X Unified audio–visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio–video synthesis to music–dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music–dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music–dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio–visual models (e.g., Veo 3, Sora 2) produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music–dance models that explicitly optimize rhythmic and kinetic coherence. The code is available at https://github.com/MM-Speech/TMD-Bench.
APA
Yang, X., Zhang, M., Pan, C., Huang, N., Yuguang, Y., Zhuo, F., Zhou, P., Zhou, J., Shan, S., Yang, S., Yang, M., You, Y. & Zhao, Z.. (2026). TMD-Bench: A Multi-Level Evaluation Paradigm for Music–Dance Co-Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:147062-147084 Available from https://proceedings.mlr.press/v306/yang26dl.html.

Related Material