Instruction Decomposition and Action Alignment for Vision-and-Language Navigation

Zihao Xin, Wentong Li, Yixuan Jiang, Bin Wang, Piji Li, Jianke Zhu, Jie Qin, Sheng-Jun Huang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:139919-139934, 2026.

Abstract

Vision-and-Language Navigation (VLN) empowered by Multimodal Large Language Models (MLLMs) is promise, yet remains challenged by long-horizon tasks with complex user instructions. Existing approaches that continuously condition on full instructions incur high latency due to abundant visual tokens and exacerbates instruction interference, where irrelevant text noise induces hallucinations. To address these limitations, we propose IDEAL-VLN ( Instruction DEcomposition and Action aLignment ), a novel paradigm that reformulates navigation as a causal inference chain. We decompose the task into two sequential steps: Semantic Anchoring and Action Alignment. We adopt a Think-Before-Act mechanism that first infers the immediate semantic anchor from the global context and then generates actions conditioned solely on this anchor. This design constructs an explicit information bottleneck, suppressing spurious correlations from irrelevant instruction. Moreover, to alleviate cognitive collapse and limited exploration during training, we introduce a hierarchical correction framework that combines semantic-level thought correction with a spatially-aware adaptive intervention strategy. This strategy adjusts expert intervention probability based on geodesic distance, effectively defining a semantic safety boundary. To support this paradigm, we contribute the Instruction-Aligned Navigation Dataset containing 160K image-text pairs. Extensive experiments demonstrate that IDEAL-VLN achieves state-of-the-art performance and robustness across major benchmarks while significantly reducing inference costs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-xin26a, title = {Instruction Decomposition and Action Alignment for Vision-and-Language Navigation}, author = {Xin, Zihao and Li, Wentong and Jiang, Yixuan and Wang, Bin and Li, Piji and Zhu, Jianke and Qin, Jie and Huang, Sheng-Jun}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {139919--139934}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xin26a/xin26a.pdf}, url = {https://proceedings.mlr.press/v306/xin26a.html}, abstract = {Vision-and-Language Navigation (VLN) empowered by Multimodal Large Language Models (MLLMs) is promise, yet remains challenged by long-horizon tasks with complex user instructions. Existing approaches that continuously condition on full instructions incur high latency due to abundant visual tokens and exacerbates instruction interference, where irrelevant text noise induces hallucinations. To address these limitations, we propose IDEAL-VLN ( Instruction DEcomposition and Action aLignment ), a novel paradigm that reformulates navigation as a causal inference chain. We decompose the task into two sequential steps: Semantic Anchoring and Action Alignment. We adopt a Think-Before-Act mechanism that first infers the immediate semantic anchor from the global context and then generates actions conditioned solely on this anchor. This design constructs an explicit information bottleneck, suppressing spurious correlations from irrelevant instruction. Moreover, to alleviate cognitive collapse and limited exploration during training, we introduce a hierarchical correction framework that combines semantic-level thought correction with a spatially-aware adaptive intervention strategy. This strategy adjusts expert intervention probability based on geodesic distance, effectively defining a semantic safety boundary. To support this paradigm, we contribute the Instruction-Aligned Navigation Dataset containing 160K image-text pairs. Extensive experiments demonstrate that IDEAL-VLN achieves state-of-the-art performance and robustness across major benchmarks while significantly reducing inference costs.} }
Endnote
%0 Conference Paper %T Instruction Decomposition and Action Alignment for Vision-and-Language Navigation %A Zihao Xin %A Wentong Li %A Yixuan Jiang %A Bin Wang %A Piji Li %A Jianke Zhu %A Jie Qin %A Sheng-Jun Huang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xin26a %I PMLR %P 139919--139934 %U https://proceedings.mlr.press/v306/xin26a.html %V 306 %X Vision-and-Language Navigation (VLN) empowered by Multimodal Large Language Models (MLLMs) is promise, yet remains challenged by long-horizon tasks with complex user instructions. Existing approaches that continuously condition on full instructions incur high latency due to abundant visual tokens and exacerbates instruction interference, where irrelevant text noise induces hallucinations. To address these limitations, we propose IDEAL-VLN ( Instruction DEcomposition and Action aLignment ), a novel paradigm that reformulates navigation as a causal inference chain. We decompose the task into two sequential steps: Semantic Anchoring and Action Alignment. We adopt a Think-Before-Act mechanism that first infers the immediate semantic anchor from the global context and then generates actions conditioned solely on this anchor. This design constructs an explicit information bottleneck, suppressing spurious correlations from irrelevant instruction. Moreover, to alleviate cognitive collapse and limited exploration during training, we introduce a hierarchical correction framework that combines semantic-level thought correction with a spatially-aware adaptive intervention strategy. This strategy adjusts expert intervention probability based on geodesic distance, effectively defining a semantic safety boundary. To support this paradigm, we contribute the Instruction-Aligned Navigation Dataset containing 160K image-text pairs. Extensive experiments demonstrate that IDEAL-VLN achieves state-of-the-art performance and robustness across major benchmarks while significantly reducing inference costs.
APA
Xin, Z., Li, W., Jiang, Y., Wang, B., Li, P., Zhu, J., Qin, J. & Huang, S.. (2026). Instruction Decomposition and Action Alignment for Vision-and-Language Navigation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:139919-139934 Available from https://proceedings.mlr.press/v306/xin26a.html.

Related Material