GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting

Milos Vukadinovic, Alan C Kwan, Debiao Li, David Ouyang
Proceedings of the 3rd Machine Learning for Health Symposium, PMLR 225:594-606, 2023.

Abstract

Generative artificial intelligence can be applied to medical imaging on tasks such as privacy-preserving image generation and super-resolution and denoising of existing images. Few prior approaches have used cardiac magnetic resonance imaging (cMRI) as a modality given the complexity of videos (the addition of the temporal dimension) as well as the limited scale of publicly available datasets. We introduce GANcMRI, a generative adversarial network that can synthesize cMRI videos with physiological guidance based on latent space prompting. GANcMRI uses a StyleGAN framework to learn the latent space from individual video frames and leverages the time-dependent trajectory between end-systolic and end-diastolic frames in the latent space to predict progression and generate motion over time. We proposed various methods for modeling latent time-dependent trajectories and found that our Frame-to-frame approach generates the best motion and video quality. GANcMRI generated high-quality cMRI image frames that are indistinguishable by cardiologists, however, artifacts in video generation allow cardiologists to still recognize the difference between real and generated videos. The generated cMRI videos can be prompted to apply physiology-based adjustments which produces clinically relevant phenotypes recognizable by cardiologists. GANcMRI has many potential applications such as data augmentation, education, anomaly detection, and preoperative planning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v225-vukadinovic23a, title = {GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting}, author = {Vukadinovic, Milos and Kwan, Alan C and Li, Debiao and Ouyang, David}, booktitle = {Proceedings of the 3rd Machine Learning for Health Symposium}, pages = {594--606}, year = {2023}, editor = {Hegselmann, Stefan and Parziale, Antonio and Shanmugam, Divya and Tang, Shengpu and Asiedu, Mercy Nyamewaa and Chang, Serina and Hartvigsen, Tom and Singh, Harvineet}, volume = {225}, series = {Proceedings of Machine Learning Research}, month = {10 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v225/vukadinovic23a/vukadinovic23a.pdf}, url = {https://proceedings.mlr.press/v225/vukadinovic23a.html}, abstract = {Generative artificial intelligence can be applied to medical imaging on tasks such as privacy-preserving image generation and super-resolution and denoising of existing images. Few prior approaches have used cardiac magnetic resonance imaging (cMRI) as a modality given the complexity of videos (the addition of the temporal dimension) as well as the limited scale of publicly available datasets. We introduce GANcMRI, a generative adversarial network that can synthesize cMRI videos with physiological guidance based on latent space prompting. GANcMRI uses a StyleGAN framework to learn the latent space from individual video frames and leverages the time-dependent trajectory between end-systolic and end-diastolic frames in the latent space to predict progression and generate motion over time. We proposed various methods for modeling latent time-dependent trajectories and found that our Frame-to-frame approach generates the best motion and video quality. GANcMRI generated high-quality cMRI image frames that are indistinguishable by cardiologists, however, artifacts in video generation allow cardiologists to still recognize the difference between real and generated videos. The generated cMRI videos can be prompted to apply physiology-based adjustments which produces clinically relevant phenotypes recognizable by cardiologists. GANcMRI has many potential applications such as data augmentation, education, anomaly detection, and preoperative planning.} }
Endnote
%0 Conference Paper %T GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting %A Milos Vukadinovic %A Alan C Kwan %A Debiao Li %A David Ouyang %B Proceedings of the 3rd Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2023 %E Stefan Hegselmann %E Antonio Parziale %E Divya Shanmugam %E Shengpu Tang %E Mercy Nyamewaa Asiedu %E Serina Chang %E Tom Hartvigsen %E Harvineet Singh %F pmlr-v225-vukadinovic23a %I PMLR %P 594--606 %U https://proceedings.mlr.press/v225/vukadinovic23a.html %V 225 %X Generative artificial intelligence can be applied to medical imaging on tasks such as privacy-preserving image generation and super-resolution and denoising of existing images. Few prior approaches have used cardiac magnetic resonance imaging (cMRI) as a modality given the complexity of videos (the addition of the temporal dimension) as well as the limited scale of publicly available datasets. We introduce GANcMRI, a generative adversarial network that can synthesize cMRI videos with physiological guidance based on latent space prompting. GANcMRI uses a StyleGAN framework to learn the latent space from individual video frames and leverages the time-dependent trajectory between end-systolic and end-diastolic frames in the latent space to predict progression and generate motion over time. We proposed various methods for modeling latent time-dependent trajectories and found that our Frame-to-frame approach generates the best motion and video quality. GANcMRI generated high-quality cMRI image frames that are indistinguishable by cardiologists, however, artifacts in video generation allow cardiologists to still recognize the difference between real and generated videos. The generated cMRI videos can be prompted to apply physiology-based adjustments which produces clinically relevant phenotypes recognizable by cardiologists. GANcMRI has many potential applications such as data augmentation, education, anomaly detection, and preoperative planning.
APA
Vukadinovic, M., Kwan, A.C., Li, D. & Ouyang, D.. (2023). GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting. Proceedings of the 3rd Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 225:594-606 Available from https://proceedings.mlr.press/v225/vukadinovic23a.html.

Related Material