Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis

Shunxing Bao, Ho Hin Lee, Qi Yang, Lucas Walker Remedios, Ruining Deng, Can Cui, Leon Yichen Cai, Kaiwen Xu, Xin Yu, Sophie Chiron, Yike Li, Nathan Heath Patterson, Yaohong Wang, Jia Li, Qi Liu, Ken S. Lau, Joseph T. Roland, Lori A. Coburn, Keith T. Wilson, Bennett A. Landman, Yuankai Huo
Medical Imaging with Deep Learning, PMLR 227:1406-1422, 2024.

Abstract

Multiplex immunofluorescence (MxIF) is an advanced molecular imaging technique that can simultaneously provide biologists with multiple (i.e., more than 20) molecular markers on a single histological tissue section. Unfortunately, due to imaging restrictions, the more routinely used hematoxylin and eosin (H&E) stain is typically unavailable with MxIF on the same tissue section. As biological H&E staining is not feasible, previous efforts have been made to obtain H&E whole slide image (WSI) from MxIF via deep learning empowered virtual staining. However, the tiling effect is a long-lasting problem in high-resolution WSI-wise synthesis. The MxIF to H&E synthesis is no exception. Limited by computational resources, the cross-stain image synthesis is typically performed at the patch-level. Thus, discontinuous intensities might be visually identified along with the patch boundaries assembling all individual patches back to a WSI. In this work, we propose a deep learning based unpaired high-resolution image synthesis method to obtain virtual H&E WSIs from MxIF WSIs (each with 27 markers/stains) with reduced tiling effects. Briefly, we first extend the CycleGAN framework by adding simultaneous nuclei and mucin segmentation supervision as spatial constraints. Then, we introduce a random sliding window shifting strategy during the optimized inference stage to alleviate the tiling effects. The validation results show that our spatially constrained synthesis method achieves a 56% performance gain for the downstream cell segmentation task. The proposed inference method reduces the tiling effects by using 50% fewer computation resources without compromising performance. The proposed random sliding window inference method is a plug-and-play module, which can be generalized and used for other high-resolution WSI image synthesis applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v227-bao24a, title = {Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis}, author = {Bao, Shunxing and Lee, Ho Hin and Yang, Qi and Remedios, Lucas Walker and Deng, Ruining and Cui, Can and Cai, Leon Yichen and Xu, Kaiwen and Yu, Xin and Chiron, Sophie and Li, Yike and Patterson, Nathan Heath and Wang, Yaohong and Li, Jia and Liu, Qi and Lau, Ken S. and Roland, Joseph T. and Coburn, Lori A. and Wilson, Keith T. and Landman, Bennett A. and Huo, Yuankai}, booktitle = {Medical Imaging with Deep Learning}, pages = {1406--1422}, year = {2024}, editor = {Oguz, Ipek and Noble, Jack and Li, Xiaoxiao and Styner, Martin and Baumgartner, Christian and Rusu, Mirabela and Heinmann, Tobias and Kontos, Despina and Landman, Bennett and Dawant, Benoit}, volume = {227}, series = {Proceedings of Machine Learning Research}, month = {10--12 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v227/bao24a/bao24a.pdf}, url = {https://proceedings.mlr.press/v227/bao24a.html}, abstract = {Multiplex immunofluorescence (MxIF) is an advanced molecular imaging technique that can simultaneously provide biologists with multiple (i.e., more than 20) molecular markers on a single histological tissue section. Unfortunately, due to imaging restrictions, the more routinely used hematoxylin and eosin (H&E) stain is typically unavailable with MxIF on the same tissue section. As biological H&E staining is not feasible, previous efforts have been made to obtain H&E whole slide image (WSI) from MxIF via deep learning empowered virtual staining. However, the tiling effect is a long-lasting problem in high-resolution WSI-wise synthesis. The MxIF to H&E synthesis is no exception. Limited by computational resources, the cross-stain image synthesis is typically performed at the patch-level. Thus, discontinuous intensities might be visually identified along with the patch boundaries assembling all individual patches back to a WSI. In this work, we propose a deep learning based unpaired high-resolution image synthesis method to obtain virtual H&E WSIs from MxIF WSIs (each with 27 markers/stains) with reduced tiling effects. Briefly, we first extend the CycleGAN framework by adding simultaneous nuclei and mucin segmentation supervision as spatial constraints. Then, we introduce a random sliding window shifting strategy during the optimized inference stage to alleviate the tiling effects. The validation results show that our spatially constrained synthesis method achieves a 56% performance gain for the downstream cell segmentation task. The proposed inference method reduces the tiling effects by using 50% fewer computation resources without compromising performance. The proposed random sliding window inference method is a plug-and-play module, which can be generalized and used for other high-resolution WSI image synthesis applications.} }
Endnote
%0 Conference Paper %T Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis %A Shunxing Bao %A Ho Hin Lee %A Qi Yang %A Lucas Walker Remedios %A Ruining Deng %A Can Cui %A Leon Yichen Cai %A Kaiwen Xu %A Xin Yu %A Sophie Chiron %A Yike Li %A Nathan Heath Patterson %A Yaohong Wang %A Jia Li %A Qi Liu %A Ken S. Lau %A Joseph T. Roland %A Lori A. Coburn %A Keith T. Wilson %A Bennett A. Landman %A Yuankai Huo %B Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2024 %E Ipek Oguz %E Jack Noble %E Xiaoxiao Li %E Martin Styner %E Christian Baumgartner %E Mirabela Rusu %E Tobias Heinmann %E Despina Kontos %E Bennett Landman %E Benoit Dawant %F pmlr-v227-bao24a %I PMLR %P 1406--1422 %U https://proceedings.mlr.press/v227/bao24a.html %V 227 %X Multiplex immunofluorescence (MxIF) is an advanced molecular imaging technique that can simultaneously provide biologists with multiple (i.e., more than 20) molecular markers on a single histological tissue section. Unfortunately, due to imaging restrictions, the more routinely used hematoxylin and eosin (H&E) stain is typically unavailable with MxIF on the same tissue section. As biological H&E staining is not feasible, previous efforts have been made to obtain H&E whole slide image (WSI) from MxIF via deep learning empowered virtual staining. However, the tiling effect is a long-lasting problem in high-resolution WSI-wise synthesis. The MxIF to H&E synthesis is no exception. Limited by computational resources, the cross-stain image synthesis is typically performed at the patch-level. Thus, discontinuous intensities might be visually identified along with the patch boundaries assembling all individual patches back to a WSI. In this work, we propose a deep learning based unpaired high-resolution image synthesis method to obtain virtual H&E WSIs from MxIF WSIs (each with 27 markers/stains) with reduced tiling effects. Briefly, we first extend the CycleGAN framework by adding simultaneous nuclei and mucin segmentation supervision as spatial constraints. Then, we introduce a random sliding window shifting strategy during the optimized inference stage to alleviate the tiling effects. The validation results show that our spatially constrained synthesis method achieves a 56% performance gain for the downstream cell segmentation task. The proposed inference method reduces the tiling effects by using 50% fewer computation resources without compromising performance. The proposed random sliding window inference method is a plug-and-play module, which can be generalized and used for other high-resolution WSI image synthesis applications.
APA
Bao, S., Lee, H.H., Yang, Q., Remedios, L.W., Deng, R., Cui, C., Cai, L.Y., Xu, K., Yu, X., Chiron, S., Li, Y., Patterson, N.H., Wang, Y., Li, J., Liu, Q., Lau, K.S., Roland, J.T., Coburn, L.A., Wilson, K.T., Landman, B.A. & Huo, Y.. (2024). Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis. Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 227:1406-1422 Available from https://proceedings.mlr.press/v227/bao24a.html.

Related Material