DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention

Xiaoya Tang, Bodong Zhang, Man M. Ho, Beatrice Knudsen, Tolga Tasdizen
Proceedings of The 8th International Conference on Medical Imaging with Deep Learning, PMLR 301:1495-1507, 2026.

Abstract

Despite the widespread adoption of transformers in medical applications, the exploration of multi-scale learning through transformers remains limited, while hierarchical representations are thought to be advantageous for medical diagnosis. We propose a novel hierarchical transformer model that adeptly integrates the feature extraction capabilities of Convolutional Neural Networks (CNNs) with the advanced representational potential of Vision Transformers (ViTs). Addressing the lack of inductive biases and dependence on extensive training datasets in ViTs, our model employs a CNN backbone to generate hierarchical visual representations. These representations are adapted for transformer input through an innovative patch tokenization process, preserving the inherited multi-scale inductive biases. We also introduce a scale-wise attention mechanism that directly captures intra-scale and inter-scale associations. This mechanism complements patch-wise attention by enhancing spatial understanding and preserving global perception, which we refer to as local and global attention, respectively. Our model significantly outperforms baseline models in terms of classification accuracy, demonstrating its efficiency in bridging the gap between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The components are designed as plug-and-play for different CNN architectures and can be adapted for multiple applications. The code is available at \href{https://github.com/xiaoyatang/DuoFormer.git}{https://github.com/xiaoyatang/DuoFormer.git}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v301-tang26a, title = {DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention}, author = {Tang, Xiaoya and Zhang, Bodong and Ho, Man M. and Knudsen, Beatrice and Tasdizen, Tolga}, booktitle = {Proceedings of The 8th International Conference on Medical Imaging with Deep Learning}, pages = {1495--1507}, year = {2026}, editor = {Tasdizen, Tolga and Elhabian, Shireen and Summers, Ronald and Chen, Chen and Koch, Lisa and Zhuang, Yan}, volume = {301}, series = {Proceedings of Machine Learning Research}, month = {09--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v301/main/assets/tang26a/tang26a.pdf}, url = {https://proceedings.mlr.press/v301/tang26a.html}, abstract = {Despite the widespread adoption of transformers in medical applications, the exploration of multi-scale learning through transformers remains limited, while hierarchical representations are thought to be advantageous for medical diagnosis. We propose a novel hierarchical transformer model that adeptly integrates the feature extraction capabilities of Convolutional Neural Networks (CNNs) with the advanced representational potential of Vision Transformers (ViTs). Addressing the lack of inductive biases and dependence on extensive training datasets in ViTs, our model employs a CNN backbone to generate hierarchical visual representations. These representations are adapted for transformer input through an innovative patch tokenization process, preserving the inherited multi-scale inductive biases. We also introduce a scale-wise attention mechanism that directly captures intra-scale and inter-scale associations. This mechanism complements patch-wise attention by enhancing spatial understanding and preserving global perception, which we refer to as local and global attention, respectively. Our model significantly outperforms baseline models in terms of classification accuracy, demonstrating its efficiency in bridging the gap between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The components are designed as plug-and-play for different CNN architectures and can be adapted for multiple applications. The code is available at \href{https://github.com/xiaoyatang/DuoFormer.git}{https://github.com/xiaoyatang/DuoFormer.git}.} }
Endnote
%0 Conference Paper %T DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention %A Xiaoya Tang %A Bodong Zhang %A Man M. Ho %A Beatrice Knudsen %A Tolga Tasdizen %B Proceedings of The 8th International Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2026 %E Tolga Tasdizen %E Shireen Elhabian %E Ronald Summers %E Chen Chen %E Lisa Koch %E Yan Zhuang %F pmlr-v301-tang26a %I PMLR %P 1495--1507 %U https://proceedings.mlr.press/v301/tang26a.html %V 301 %X Despite the widespread adoption of transformers in medical applications, the exploration of multi-scale learning through transformers remains limited, while hierarchical representations are thought to be advantageous for medical diagnosis. We propose a novel hierarchical transformer model that adeptly integrates the feature extraction capabilities of Convolutional Neural Networks (CNNs) with the advanced representational potential of Vision Transformers (ViTs). Addressing the lack of inductive biases and dependence on extensive training datasets in ViTs, our model employs a CNN backbone to generate hierarchical visual representations. These representations are adapted for transformer input through an innovative patch tokenization process, preserving the inherited multi-scale inductive biases. We also introduce a scale-wise attention mechanism that directly captures intra-scale and inter-scale associations. This mechanism complements patch-wise attention by enhancing spatial understanding and preserving global perception, which we refer to as local and global attention, respectively. Our model significantly outperforms baseline models in terms of classification accuracy, demonstrating its efficiency in bridging the gap between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The components are designed as plug-and-play for different CNN architectures and can be adapted for multiple applications. The code is available at \href{https://github.com/xiaoyatang/DuoFormer.git}{https://github.com/xiaoyatang/DuoFormer.git}.
APA
Tang, X., Zhang, B., Ho, M.M., Knudsen, B. & Tasdizen, T.. (2026). DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention. Proceedings of The 8th International Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 301:1495-1507 Available from https://proceedings.mlr.press/v301/tang26a.html.

Related Material