NeurIPS 2025 Foundation Models Meet Embodied Agents Challenge: Systems, Analysis, and Lessons Learned

Tianwei Bao, Qineng Wang, Kangrui Wang, Bryan Yu Zhou, Tony Lee, Li Erran Li, Weiyu Liu, Ruohan Zhang, Yejin Choi, Percy Liang, Jiayuan Mao, Li Fei-Fei, Jiajun Wu, Manling Li
Proceedings of the NeurIPS 2023-2025 Competition Track, PMLR 276:1-16, 2026.

Abstract

While Large Language Models (LLMs) have demonstrated impressive high-level reasoning capabilities, their deployment in embodied decision-making tasks remains hindered by a lack of standardized, diagnostic evaluations. We present a retrospective on the NeurIPS 2025 Embodied Agent Interface Challenge, which systematically benchmarked LLMs across four critical reasoning modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. Hosted on the BEHAVIOR and VirtualHome simulators, the competition departed from traditional, binary task-success metrics. Instead, it leveraged Linear Temporal Logic (LTL) to evaluate alternative viable trajectories and introduced fine-grained metrics to pinpoint specific failures, such as affordance violations, hallucinations, and missing preconditions. The competition attracted significant engagement globally, with 58 registered teams developing diverse solutions. An analysis of the winning and most innovative submissions reveals a clear trend: while hybrid neuro-symbolic pipelines and test-time compute significantly enhance trajectory feasibility, spatial relationship modeling remains a pervasive bottleneck for current LLMs. In this report, we detail the competition design, summarize the top-performing methodologies, and present key lessons learned to guide the future development of robust, interpretable embodied AI systems.

Cite this Paper


BibTeX
@InProceedings{pmlr-v276-bao26a, title = {NeurIPS 2025 Foundation Models Meet Embodied Agents Challenge: Systems, Analysis, and Lessons Learned}, author = {Bao, Tianwei and Wang, Qineng and Wang, Kangrui and Zhou, Bryan Yu and Lee, Tony and Li, Li Erran and Liu, Weiyu and Zhang, Ruohan and Choi, Yejin and Liang, Percy and Mao, Jiayuan and Fei-Fei, Li and Wu, Jiajun and Li, Manling}, booktitle = {Proceedings of the NeurIPS 2023-2025 Competition Track}, pages = {1--16}, year = {2026}, editor = {Qin, Tao and Zhang, Kun and Frellsen, Jes}, volume = {276}, series = {Proceedings of Machine Learning Research}, month = {02--07 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v276/main/assets/bao26a/bao26a.pdf}, url = {https://proceedings.mlr.press/v276/bao26a.html}, abstract = {While Large Language Models (LLMs) have demonstrated impressive high-level reasoning capabilities, their deployment in embodied decision-making tasks remains hindered by a lack of standardized, diagnostic evaluations. We present a retrospective on the NeurIPS 2025 Embodied Agent Interface Challenge, which systematically benchmarked LLMs across four critical reasoning modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. Hosted on the BEHAVIOR and VirtualHome simulators, the competition departed from traditional, binary task-success metrics. Instead, it leveraged Linear Temporal Logic (LTL) to evaluate alternative viable trajectories and introduced fine-grained metrics to pinpoint specific failures, such as affordance violations, hallucinations, and missing preconditions. The competition attracted significant engagement globally, with 58 registered teams developing diverse solutions. An analysis of the winning and most innovative submissions reveals a clear trend: while hybrid neuro-symbolic pipelines and test-time compute significantly enhance trajectory feasibility, spatial relationship modeling remains a pervasive bottleneck for current LLMs. In this report, we detail the competition design, summarize the top-performing methodologies, and present key lessons learned to guide the future development of robust, interpretable embodied AI systems.} }
Endnote
%0 Conference Paper %T NeurIPS 2025 Foundation Models Meet Embodied Agents Challenge: Systems, Analysis, and Lessons Learned %A Tianwei Bao %A Qineng Wang %A Kangrui Wang %A Bryan Yu Zhou %A Tony Lee %A Li Erran Li %A Weiyu Liu %A Ruohan Zhang %A Yejin Choi %A Percy Liang %A Jiayuan Mao %A Li Fei-Fei %A Jiajun Wu %A Manling Li %B Proceedings of the NeurIPS 2023-2025 Competition Track %C Proceedings of Machine Learning Research %D 2026 %E Tao Qin %E Kun Zhang %E Jes Frellsen %F pmlr-v276-bao26a %I PMLR %P 1--16 %U https://proceedings.mlr.press/v276/bao26a.html %V 276 %X While Large Language Models (LLMs) have demonstrated impressive high-level reasoning capabilities, their deployment in embodied decision-making tasks remains hindered by a lack of standardized, diagnostic evaluations. We present a retrospective on the NeurIPS 2025 Embodied Agent Interface Challenge, which systematically benchmarked LLMs across four critical reasoning modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. Hosted on the BEHAVIOR and VirtualHome simulators, the competition departed from traditional, binary task-success metrics. Instead, it leveraged Linear Temporal Logic (LTL) to evaluate alternative viable trajectories and introduced fine-grained metrics to pinpoint specific failures, such as affordance violations, hallucinations, and missing preconditions. The competition attracted significant engagement globally, with 58 registered teams developing diverse solutions. An analysis of the winning and most innovative submissions reveals a clear trend: while hybrid neuro-symbolic pipelines and test-time compute significantly enhance trajectory feasibility, spatial relationship modeling remains a pervasive bottleneck for current LLMs. In this report, we detail the competition design, summarize the top-performing methodologies, and present key lessons learned to guide the future development of robust, interpretable embodied AI systems.
APA
Bao, T., Wang, Q., Wang, K., Zhou, B.Y., Lee, T., Li, L.E., Liu, W., Zhang, R., Choi, Y., Liang, P., Mao, J., Fei-Fei, L., Wu, J. & Li, M.. (2026). NeurIPS 2025 Foundation Models Meet Embodied Agents Challenge: Systems, Analysis, and Lessons Learned. Proceedings of the NeurIPS 2023-2025 Competition Track, in Proceedings of Machine Learning Research 276:1-16 Available from https://proceedings.mlr.press/v276/bao26a.html.

Related Material