GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification

Moinak Bhattacharya, Prateek Prasanna
Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning, PMLR 250:103-118, 2024.

Abstract

We present GazeDiff, a novel architecture that leverages radiologists\’{eye} gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis.

Cite this Paper

BibTeX
@InProceedings{pmlr-v250-bhattacharya24a, title = {GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification}, author = {Bhattacharya, Moinak and Prasanna, Prateek}, booktitle = {Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning}, pages = {103--118}, year = {2024}, editor = {Burgos, Ninon and Petitjean, Caroline and Vakalopoulou, Maria and Christodoulidis, Stergios and Coupe, Pierrick and Delingette, Hervé and Lartizien, Carole and Mateus, Diana}, volume = {250}, series = {Proceedings of Machine Learning Research}, month = {03--05 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v250/main/assets/bhattacharya24a/bhattacharya24a.pdf}, url = {https://proceedings.mlr.press/v250/bhattacharya24a.html}, abstract = {We present GazeDiff, a novel architecture that leverages radiologists\’{eye} gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis.} }
Endnote
%0 Conference Paper %T GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification %A Moinak Bhattacharya %A Prateek Prasanna %B Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2024 %E Ninon Burgos %E Caroline Petitjean %E Maria Vakalopoulou %E Stergios Christodoulidis %E Pierrick Coupe %E Hervé Delingette %E Carole Lartizien %E Diana Mateus %F pmlr-v250-bhattacharya24a %I PMLR %P 103--118 %U https://proceedings.mlr.press/v250/bhattacharya24a.html %V 250 %X We present GazeDiff, a novel architecture that leverages radiologists\’{eye} gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis.
APA
Bhattacharya, M. & Prasanna, P.. (2024). GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification. Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 250:103-118 Available from https://proceedings.mlr.press/v250/bhattacharya24a.html.

Related Material