GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding

Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, Caiwen Ding
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:65506-65519, 2026.

Abstract

Multimodal large language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight – a model trained for image-grounded reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy, surpassing V2P-7B (50.6% with 9.6M training samples) and GTA-1-7B (50.1% with 1.56M training samples).

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-lei26h, title = {{GUI}-Spotlight: Adaptive Iterative Focus Refinement for Enhanced {GUI} Visual Grounding}, author = {Lei, Bin and Xu, Nuo and Payani, Ali and Hong, Mingyi and Liao, Chunhua and Cao, Yu and Ding, Caiwen}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {65506--65519}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lei26h/lei26h.pdf}, url = {https://proceedings.mlr.press/v306/lei26h.html}, abstract = {Multimodal large language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight – a model trained for image-grounded reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy, surpassing V2P-7B (50.6% with 9.6M training samples) and GTA-1-7B (50.1% with 1.56M training samples).} }
Endnote
%0 Conference Paper %T GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding %A Bin Lei %A Nuo Xu %A Ali Payani %A Mingyi Hong %A Chunhua Liao %A Yu Cao %A Caiwen Ding %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lei26h %I PMLR %P 65506--65519 %U https://proceedings.mlr.press/v306/lei26h.html %V 306 %X Multimodal large language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight – a model trained for image-grounded reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy, surpassing V2P-7B (50.6% with 9.6M training samples) and GTA-1-7B (50.1% with 1.56M training samples).
APA
Lei, B., Xu, N., Payani, A., Hong, M., Liao, C., Cao, Y. & Ding, C.. (2026). GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:65506-65519 Available from https://proceedings.mlr.press/v306/lei26h.html.

Related Material