Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency

Jie Yang, Ye Zhu, Chaoqun Wang, Zhen Li, Ruimao Zhang
Medical Imaging with Deep Learning, PMLR 227:1602-1622, 2024.

Abstract

Integrating multi-modal data to promote medical image analysis has recently gained great attention. This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired multi-modal medical images. Our approach tackles two critical issues of this task from a practical perspective: (1) how to effectively learn the semantic consistencies of various modalities (e.g., CT and MRI), and (2) how to leverage the above consistencies to regularize the network learning while preserving its simplicity. To address (1), we leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. To solve (2), the proposed EAM is designed as an external plug-and-play one, which can be discarded once the model is optimized. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios: (1) cardiac structure segmentation, and (2) abdominal multi-organ segmentation. Extensive results show that the proposed method outperforms its counterparts by a wide margin.

Cite this Paper

BibTeX
@InProceedings{pmlr-v227-yang24c, title = {Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency}, author = {Yang, Jie and Zhu, Ye and Wang, Chaoqun and Li, Zhen and Zhang, Ruimao}, booktitle = {Medical Imaging with Deep Learning}, pages = {1602--1622}, year = {2024}, editor = {Oguz, Ipek and Noble, Jack and Li, Xiaoxiao and Styner, Martin and Baumgartner, Christian and Rusu, Mirabela and Heinmann, Tobias and Kontos, Despina and Landman, Bennett and Dawant, Benoit}, volume = {227}, series = {Proceedings of Machine Learning Research}, month = {10--12 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v227/yang24c/yang24c.pdf}, url = {https://proceedings.mlr.press/v227/yang24c.html}, abstract = {Integrating multi-modal data to promote medical image analysis has recently gained great attention. This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired multi-modal medical images. Our approach tackles two critical issues of this task from a practical perspective: (1) how to effectively learn the semantic consistencies of various modalities (e.g., CT and MRI), and (2) how to leverage the above consistencies to regularize the network learning while preserving its simplicity. To address (1), we leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. To solve (2), the proposed EAM is designed as an external plug-and-play one, which can be discarded once the model is optimized. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios: (1) cardiac structure segmentation, and (2) abdominal multi-organ segmentation. Extensive results show that the proposed method outperforms its counterparts by a wide margin.} }
Endnote
%0 Conference Paper %T Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency %A Jie Yang %A Ye Zhu %A Chaoqun Wang %A Zhen Li %A Ruimao Zhang %B Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2024 %E Ipek Oguz %E Jack Noble %E Xiaoxiao Li %E Martin Styner %E Christian Baumgartner %E Mirabela Rusu %E Tobias Heinmann %E Despina Kontos %E Bennett Landman %E Benoit Dawant %F pmlr-v227-yang24c %I PMLR %P 1602--1622 %U https://proceedings.mlr.press/v227/yang24c.html %V 227 %X Integrating multi-modal data to promote medical image analysis has recently gained great attention. This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired multi-modal medical images. Our approach tackles two critical issues of this task from a practical perspective: (1) how to effectively learn the semantic consistencies of various modalities (e.g., CT and MRI), and (2) how to leverage the above consistencies to regularize the network learning while preserving its simplicity. To address (1), we leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. To solve (2), the proposed EAM is designed as an external plug-and-play one, which can be discarded once the model is optimized. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios: (1) cardiac structure segmentation, and (2) abdominal multi-organ segmentation. Extensive results show that the proposed method outperforms its counterparts by a wide margin.
APA
Yang, J., Zhu, Y., Wang, C., Li, Z. & Zhang, R.. (2024). Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency. Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 227:1602-1622 Available from https://proceedings.mlr.press/v227/yang24c.html.

Related Material