UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis

Junzhi Ning, Wei Li, Cheng Tang, Jiashi Lin, Chenglong Ma, Chaoyang Zhang, Jiyao Liu, Ying Chen, Shujian Gao, Yuandong Pu, Huihui Xu, Chenhui Gou, Ziyan Huang, Yi Xin, Qi Qin, Diping Song, Bin Fu, Guang Yang, Yuanfeng Ji, Tianbin Li, Yanzhou Su, Jin Ye, Shixiang Tang, Zhongying Deng, Lihao Liu, Ming Hu, Junjun He
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:93376-93421, 2026.

Abstract

Medical workflows routinely combine reading images with producing visual and textual outputs, making both image understanding and generation central to medical AI. Most existing systems, however, address these abilities in isolated models, losing the shared knowledge that a unified architecture could exploit. To bridge this gap, we present UniMedVL, the first unified medical model that seamlessly integrates multimodal understanding and generation capabilities within a single model without switching weights. We achieve this via a tailored progressive training pipeline where understanding and generation mutually reinforce each other. To effectively train UniMedVL, we curate UniMedVL-5M, the first large-scale medical dataset comprising over 5.6M instances across 8 medical imaging modalities, tailored for multimodal input-output tasks in unified medical understanding and generation. Experimental results demonstrate that UniMedVL achieves competitive performance on five medical understanding benchmarks. Crucially, UniMedVL natively supports diverse interleaved generation tasks, e.g., virtual staining, super-resolution, cross-modal synthesis, essential for complex medical workflows. Our code and dataset are publicly available.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ning26d, title = {{U}ni{M}ed{VL}: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis}, author = {Ning, Junzhi and Li, Wei and Tang, Cheng and Lin, Jiashi and Ma, Chenglong and Zhang, Chaoyang and Liu, Jiyao and Chen, Ying and Gao, Shujian and Pu, Yuandong and Xu, Huihui and Gou, Chenhui and Huang, Ziyan and Xin, Yi and Qin, Qi and Song, Diping and Fu, Bin and Yang, Guang and Ji, Yuanfeng and Li, Tianbin and Su, Yanzhou and Ye, Jin and Tang, Shixiang and Deng, Zhongying and Liu, Lihao and Hu, Ming and He, Junjun}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {93376--93421}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ning26d/ning26d.pdf}, url = {https://proceedings.mlr.press/v306/ning26d.html}, abstract = {Medical workflows routinely combine reading images with producing visual and textual outputs, making both image understanding and generation central to medical AI. Most existing systems, however, address these abilities in isolated models, losing the shared knowledge that a unified architecture could exploit. To bridge this gap, we present UniMedVL, the first unified medical model that seamlessly integrates multimodal understanding and generation capabilities within a single model without switching weights. We achieve this via a tailored progressive training pipeline where understanding and generation mutually reinforce each other. To effectively train UniMedVL, we curate UniMedVL-5M, the first large-scale medical dataset comprising over 5.6M instances across 8 medical imaging modalities, tailored for multimodal input-output tasks in unified medical understanding and generation. Experimental results demonstrate that UniMedVL achieves competitive performance on five medical understanding benchmarks. Crucially, UniMedVL natively supports diverse interleaved generation tasks, e.g., virtual staining, super-resolution, cross-modal synthesis, essential for complex medical workflows. Our code and dataset are publicly available.} }
Endnote
%0 Conference Paper %T UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis %A Junzhi Ning %A Wei Li %A Cheng Tang %A Jiashi Lin %A Chenglong Ma %A Chaoyang Zhang %A Jiyao Liu %A Ying Chen %A Shujian Gao %A Yuandong Pu %A Huihui Xu %A Chenhui Gou %A Ziyan Huang %A Yi Xin %A Qi Qin %A Diping Song %A Bin Fu %A Guang Yang %A Yuanfeng Ji %A Tianbin Li %A Yanzhou Su %A Jin Ye %A Shixiang Tang %A Zhongying Deng %A Lihao Liu %A Ming Hu %A Junjun He %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ning26d %I PMLR %P 93376--93421 %U https://proceedings.mlr.press/v306/ning26d.html %V 306 %X Medical workflows routinely combine reading images with producing visual and textual outputs, making both image understanding and generation central to medical AI. Most existing systems, however, address these abilities in isolated models, losing the shared knowledge that a unified architecture could exploit. To bridge this gap, we present UniMedVL, the first unified medical model that seamlessly integrates multimodal understanding and generation capabilities within a single model without switching weights. We achieve this via a tailored progressive training pipeline where understanding and generation mutually reinforce each other. To effectively train UniMedVL, we curate UniMedVL-5M, the first large-scale medical dataset comprising over 5.6M instances across 8 medical imaging modalities, tailored for multimodal input-output tasks in unified medical understanding and generation. Experimental results demonstrate that UniMedVL achieves competitive performance on five medical understanding benchmarks. Crucially, UniMedVL natively supports diverse interleaved generation tasks, e.g., virtual staining, super-resolution, cross-modal synthesis, essential for complex medical workflows. Our code and dataset are publicly available.
APA
Ning, J., Li, W., Tang, C., Lin, J., Ma, C., Zhang, C., Liu, J., Chen, Y., Gao, S., Pu, Y., Xu, H., Gou, C., Huang, Z., Xin, Y., Qin, Q., Song, D., Fu, B., Yang, G., Ji, Y., Li, T., Su, Y., Ye, J., Tang, S., Deng, Z., Liu, L., Hu, M. & He, J.. (2026). UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:93376-93421 Available from https://proceedings.mlr.press/v306/ning26d.html.

Related Material