
- title: 'NeurIPS 2025 Foundation Models Meet Embodied Agents Challenge: Systems, Analysis, and Lessons Learned'
  abstract: 'While Large Language Models (LLMs) have demonstrated impressive high-level reasoning capabilities, their deployment in embodied decision-making tasks remains hindered by a lack of standardized, diagnostic evaluations. We present a retrospective on the NeurIPS 2025 Embodied Agent Interface Challenge, which systematically benchmarked LLMs across four critical reasoning modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. Hosted on the BEHAVIOR and VirtualHome simulators, the competition departed from traditional, binary task-success metrics. Instead, it leveraged Linear Temporal Logic (LTL) to evaluate alternative viable trajectories and introduced fine-grained metrics to pinpoint specific failures, such as affordance violations, hallucinations, and missing preconditions. The competition attracted significant engagement globally, with 58 registered teams developing diverse solutions. An analysis of the winning and most innovative submissions reveals a clear trend: while hybrid neuro-symbolic pipelines and test-time compute significantly enhance trajectory feasibility, spatial relationship modeling remains a pervasive bottleneck for current LLMs. In this report, we detail the competition design, summarize the top-performing methodologies, and present key lessons learned to guide the future development of robust, interpretable embodied AI systems.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/bao26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/bao26a/bao26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-bao26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Tianwei
    family: Bao
  - given: Qineng
    family: Wang
  - given: Kangrui
    family: Wang
  - given: Bryan Yu
    family: Zhou
  - given: Tony
    family: Lee
  - given: Li Erran
    family: Li
  - given: Weiyu
    family: Liu
  - given: Ruohan
    family: Zhang
  - given: Yejin
    family: Choi
  - given: Percy
    family: Liang
  - given: Jiayuan
    family: Mao
  - given: Li
    family: Fei-Fei
  - given: Jiajun
    family: Wu
  - given: Manling
    family: Li
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 1-16
  id: bao26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 1
  lastpage: 16
  published: 2026-11-21 00:00:00 +0000
- title: 'MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark'
  abstract: 'We present the results of the MMU-RAGent competition. The competition tasked participants with building systems capable of answering real-world queries drawn from MS MARCO Web Search and Chatbot Arena conversations, as well as queries collected live through the RAG-Arena evaluation platform. To support large-scale retrieval, participants were given API access to the English subsets of ClueWeb22-B and ClueWeb22-A (87M and 800M documents) and AWS-hosted infrastructure for system deployment and evaluation. Submissions were evaluated using a combination of human Likert-scale ratings, live preference judgments collected via RAG-Arena, LLM-as-a-Judge evaluation, and automatic metrics. The competition allowed both open-source and closed-source approaches, including systems leveraging proprietary search APIs or models, with separate leaderboard categories to ensure transparent comparison. Participating systems explored a wide range of RAG architectures, including hierarchical retrieval pipelines, structured document parsing, multi-stage reranking, and iterative generation with citation verification. The competition provides a new benchmark for studying retrieval-augmented generation under realistic deployment conditions and highlights emerging system design patterns for scalable and user-aligned RAG.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/chan26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/chan26a/chan26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-chan26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Luo Qi
    family: Chan
  - given: Tevin
    family: Wang
  - given: Shuting
    family: Wang
  - given: Zhihan
    family: Zhang
  - given: Alfredo
    family: Gomez
  - given: Prahaladh
    family: Chandrahasan
  - given: Andy
    family: Tang
  - given: Lan
    family: Yan
  - given: Zimeng
    family: Qiu
  - given: Morteza
    family: Ziyadi
  - given: Tongshuang
    family: Wu
  - given: Mona
    family: Diab
  - given: Akari
    family: Asai
  - given: Chenyan
    family: Xiong
  - given: Izat
    family: Temiraliev
  - given: Serene
    family: Cheng
  - given: Diji
    family: Yang
  - given: Yi
    family: Zhang
  - given: Kun
    family: Ran
  - given: Marwah
    family: Alaofi
  - given: Danula
    family: Hettiachchi
  - given: Chenglong
    family: Ma
  - given: Khoi Nguyen
    family: Dinh Anh
  - given: Khoi Vo
    family: Nguyen
  - given: Sachin
    family: Pathiyan Cherumanal
  - given: Lida
    family: Rashidi
  - given: Falk
    family: Scholer
  - given: Damiano
    family: Spina
  - given: Shuoqi
    family: Sun
  - given: Oleg
    family: Zendel
  - given: Ikuya
    family: Yamada
  - given: Wataru
    family: Ikeda
  - given: Ko
    family: Yoshida
  - given: Mengyu
    family: Ye
  - given: Hinata
    family: Sugimoto
  - given: Masatoshi
    family: Suzuki
  - given: Hisanori
    family: Ozaki
  - given: Jun
    family: Suzuki
  - given: Cattalyya
    family: Nuengsigkapian
  - given: Dmitrii
    family: Magas
  - given: Quentin
    family: Fever
  - given: Naziha
    family: Aslam
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 17-52
  id: chan26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 17
  lastpage: 52
  published: 2026-11-21 00:00:00 +0000
- title: 'DCVLR: Data Curation for Vision Language Reasoning Competition'
  abstract: 'At NeurIPS 2025, we presented the first data-centric, open-data and open-models competition for data curation in vision-language reasoning. The competition focused on improving visual reasoning capabilities of vision-language models (VLMs) through data curation under a fixed supervised-finetuning (SFT) configuration and budget. Submissions were evaluated by fine-tuning a fixed VLM on the curated datasets and measuring performance on reasoning benchmarks of multiple-choice visual reasoning questions spanning diverse sources. The competition provided all necessary resources, including fine-tuning scripts, evaluation code, and baselines. Additionally, as a first of competitions of this kind, organizers also ran the entire SFT and evaluation pipeline on behalf of the contestants, thereby removing a key barrier to entry present in previous data-centric competitions. The result was the largest data-centric competition to date, with 130 signups and 20 submissions. DCVLR centered data curation as the key variable for scientific progress in VLM adaptation, highlighting its importance for real-world applications such as education, biomedicine, and scientific reasoning.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/elachqar26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/elachqar26a/elachqar26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-elachqar26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Oussama
    family: Elachqar
  - given: Benjamin
    family: Feuer
  - given: Neha
    family: Hulkund
  - given: Rohun
    family: Tripathi
  - given: Rohun
    family: Agrawal
  - given: Thao
    family: Nguyen
  - given: Nimrod
    family: Shabtay
  - given: Vishaal
    family: Udandarao
  - given: Xiaohan
    family: Wang
  - given: Stefan
    family: Webb
  - given: Yuhui
    family: Zhang
  - given: Sara
    family: Beery
  - given: Georgia
    family: Gkioxari
  - given: Paul
    family: Liang
  - given: Ludwig
    family: Schmidt
  - given: Saining
    family: Xie
  - given: Serena
    family: Yeung-Levy
  - given: Michael
    family: Buriek
  - given: Pavel
    family: Bushuyeu
  - given: Dimitrios
    family: Damianos
  - given: Eleni
    family: Kasoura
  - given: Vassilis
    family: Katsouros
  - given: Vikas
    family: Kumar
  - given: Igor
    family: Molybog
  - given: Georgios
    family: Paraskevopoulos
  - given: Yosub
    family: Shin
  - given: Boris
    family: Sobolev
  - given: Samuel
    family: Watson
  - given: Haoyang
    family: Xu
  - given: Emmanouil
    family: Koukoumidis
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 53-68
  id: elachqar26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 53
  lastpage: 68
  published: 2026-11-21 00:00:00 +0000
- title: 'Benchmarking Non-Invasive Speech BCIs: Lessons Learned from the 2025 PNPL Competition'
  abstract: 'We present a retrospective on the 2025 PNPL Competition, an open machine-learning benchmark for decoding speech from non-invasive brain recordings, which ran from June to September 2025 and culminated in a workshop at NeurIPS that December. The competition used the LibriBrain dataset — over 50 hours of within-subject MEG with standard data splits — and challenged participants to tackle two tasks: Speech Detection (binary frame-level classification) and Phoneme Classification (39-class ARPAbet prediction). Across 155 registered teams and 6,041 submissions, top F1-macro scores reached 95.6% for Speech Detection and 73.6% for Phoneme Classification, exceeding prior baselines of 68% and 44%, respectively, by wide margins. High-performing submissions converged on a common architectural template — temporal CNN or TCN backbones with task-specific recurrent or attention-based heads — whilst task-specific strategies such as temporal smoothing and ensemble voting proved critical for competitive performance. Hidden track evaluations suggest that main-track Speech Detection models generalised poorly to balanced class distributions. This could imply a reliance on class priors rather than robust neural decoding, highlighting a useful lesson: evaluation distributions should reflect intended deployment conditions. We further reflect on design choices, organisational lessons, and plans for future competitions in this series.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/elvers26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/elvers26a/elvers26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-elvers26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Gereon
    family: Elvers
  - given: Gilad
    family: Landau
  - given: Francesco
    family: Mantegna
  - given: Miran
    family: Özdogan
  - given: Tasha
    family: Kim
  - given: Teyun
    family: Kwon
  - given: SungJun
    family: Cho
  - given: Benjamin
    family: Ballyk
  - given: Luisa
    family: Kurth
  - given: Dulhan
    family: Jayalath
  - given: Pratik
    family: Somaiya
  - given: Chetan
    family: Gohil
  - given: Brendan
    family: Shillingford
  - given: Greg
    family: Farquhar
  - given: Minqi
    family: Jiang
  - given: Caglar
    family: Gulcehre
  - given: Xabier
    prefix: de
    family: Zuazo
  - given: Hamza
    family: Abdelhedi
  - given: Yorguin
    family: Mantilla Ramos
  - given: Karim
    family: Jerbi
  - given: Mark
    family: Woolrich
  - given: Natalie
    family: Voets
  - given: Oiwi
    family: Parker Jones
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 69-89
  id: elvers26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 69
  lastpage: 89
  published: 2026-11-21 00:00:00 +0000
- title: 'Weather4cast at NeurIPS 2023 – Data Fusion for Quantitative Hi-Res Rain Movie Prediction under Spatio-temporal Shifts'
  abstract: 'Now in its third year, Weather4cast 2023 again advanced modern algorithms in AI and machine learning through a highly topical interdisciplinary challenge at the NeurIPS Competition Track. Predicting hi-res rain radar movies from multi-band satellite sensors required data fusion of complementary signal sources and video frame prediction with super-resolution techniques. This year, quantitative prediction of rain rates was assessed using a balanced multi-class metric. Accurate rain predictions are ever more critical for everyone, with climate change increasing the frequency of extreme precipitation events. Weather4cast models thus have a particular impact for the many regions on Earth where costly weather radar data are not available. We here present highlights and insights emerging from the competition. To reward models extracting relevant mechanistic patterns reflecting the underlying complex weather systems, our evaluation tested their responses to spatio-temporal shifts. Algorithms needed to forecast 4 h or 8 h of ground-based hi-res precipitation radar from lo-res satellite spectral images in a unique cross-sensor prediction challenge, and were evaluated within and across a variety of regions on Earth with diverse climate and different distributions of heavy precipitation events. Robustness to distributional shifts over time was achieved by testing predictions on data one year after the training period. Notably, the capability of models to correctly capture spatio-temporal patterns qualitatively varied considerably and was not always in line with leaderboard scores. This suggests that more sophisticated metrics will be required to accurately reflect the effectiveness of AI models in corresponding meteorological applications. Weather4cast will continue to explore the expansive data set and advance benchmark challenges in the field en route to validated foundation models for weather.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/gruca26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/gruca26a/gruca26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-gruca26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Aleksandra
    family: Gruca
  - given: Xavier
    family: Calbet
  - given: Pilar
    family: Rípodas
  - given: Llorenç
    family: Lliso
  - given: Federico
    family: Serva
  - given: Bertrand
    family: Le Saux
  - given: Lu
    family: Han
  - given: Xu-Yang
    family: Chen
  - given: Han-Jia
    family: Ye
  - given: De-Chuan
    family: Zhan
  - given: P. Jyoteeshkumar
    family: Reddy
  - given: Harish
    family: Baki
  - given: Sandeep
    family: Chinta
  - given: Pablo
    family: Izquierdo Ayala
  - given: Akshay
    family: Punjabi
  - given: Ajitabh
    family: Kumar
  - given: Rafael
    family: Pablos Sarabia
  - given: Ira
    family: Assent
  - given: Joachim
    family: Nyborg
  - given: Morten
    family: Birk
  - given: David P.
    family: Kreil
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 90-105
  id: gruca26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 90
  lastpage: 105
  published: 2026-11-21 00:00:00 +0000
- title: 'Weather4cast at NeurIPS 2025 – Multi-task Challenges for Weather and Pollution Pattern Prediction on the Road to Hi-Res Foundation Models'
  abstract: 'The Weather4cast competition series aims to advance AI and machine learning algorithms through a highly topical interdisciplinary challenge (www.weather4cast.net). The core task involves predicting high-resolution rain radar movies from multi-band satellite sensors, which requires the application of data fusion across complementary signal sources, multi-channel video frame prediction, and super-resolution techniques. To evaluate the models’ ability to extract relevant mechanistic patterns, the competition incorporates spatio-temporal shifts, evaluating predictions both within and across diverse Earth regions with varying climates and precipitation distributions. Now, in its fifth year, Weather4cast moves to improve forecasts world-wide on an expansive data set with over a magnitude more hi-res rain radar data, allowing a move towards Foundation Models through multi-modality, multi-scale, multi-task challenges. Accurate rain predictions are becoming ever more critical for everyone, with climate change increasing the frequency of extreme precipitation events. Notably, the new models and insights will have a particular impact for the many regions on Earth where costly weather radar data are not available. As a complementary application-specific forecasting endpoint, in 2025, for the first time, we added a pollution forecasting challenge.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/gruca26b.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/gruca26b/gruca26b.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-gruca26b.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Aleksandra
    family: Gruca
  - given: Xavier
    family: Calbet
  - given: Pilar
    family: Rípodas
  - given: Llorenç
    family: Lliso
  - given: Federico
    family: Serva
  - given: Levi
    family: Harris
  - given: Tianlong
    family: Chen
  - given: Luciano Araujo
    family: Dourado Filho
  - given: Almir Moreira
    prefix: da
    family: Silva Neto
  - given: Anthony
    family: Miyaguchi
  - given: Rodrigo Pereira
    family: David
  - given: Rodrigo Tripodi
    family: Calumby
  - given: Lukas
    family: Picek
  - given: Kaushik
    family: Gopalan
  - given: Atharva
    family: Deshpande
  - given: Jeet
    family: Shah
  - given: Hrishikesh
    family: Simu
  - given: Ansh
    family: Kushwaha
  - given: Anushree
    family: Bhuskute
  - given: David P.
    family: Kreil
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 106-122
  id: gruca26b
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 106
  lastpage: 122
  published: 2026-11-21 00:00:00 +0000
- title: 'The 2025 Google Code Golf Championship'
  abstract: 'For over half a decade, the Abstraction and Reasoning Corpus has played a critical role in evaluating the fluid intelligence and skill acquisition efficiency of machine learning systems. While the dataset has proven especially useful in advancing program synthesis, few investments have been made in distilling the transformational logic of these tasks into their most irreducible computational forms. In conjunction with the NeurIPS 2025 Competition Track, we organized the 2025 Google Code Golf Championship, a three-month online challenge designed to elicit minimal Python solutions for all training puzzles in the seminal ARC-AGI-1 benchmark suite. Sponsored by Google DeepMind and funded with $100,000 in prizes, the contest attracted over 1,000 teams and yielded tens of thousands of verified, open-source programs that collectively establish tight upper bounds on the Kolmogorov complexity of each task. In this paper, we offer a comprehensive retrospective of the event, including an overview of the contest format, a survey of approaches taken by top teams, and a discussion of potential implications for the future design of intelligent agents.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/moffitt26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/moffitt26a/moffitt26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-moffitt26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Michael D.
    family: Moffitt
  - given: Divy
    family: Thakkar
  - given: Ryan
    family: Burnell
  - given: Orhan
    family: Firat
  - given: Walter
    family: Reade
  - given: Sohier
    family: Dane
  - given: Addison
    family: Howard
  - given: Lucas
    family: Deutschmann
  - given: Luke
    family: Gustafson
  - given: Adam
    family: Kues
  - given: Mukundan
    family: Senthilkumar
  - given: Nicolas
    family: Posada
  - given: Lyndon
    family: Ho
  - given: Larry
    family: Wu
  - given: Lav
    family: Patel
  - given: Hunter
    family: Richardson
  - given: Wei Heng
    family: Kuan
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 123-173
  id: moffitt26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 123
  lastpage: 173
  published: 2026-11-21 00:00:00 +0000
- title: 'Lessons Learned from Auto-Bidding in the Large-Scale Auctions Competition at NeurIPS 2024'
  abstract: 'Decision-making in large-scale games is an essential research area in artificial intelligence, with significant real-world impact. An agent faces the critical task of making high-frequency strategic decisions in an uncertain and competitive environment. This environment is characterized by substantial randomness and rapidly changing strategies from numerous competitors. However, the shortage of large-scale, realistic game systems and datasets has hindered research progress in this area. To provide opportunities for in-depth research on this valuable problem, we have presented the Auto-Bidding in Large-Scale Auctions Competition, derived from online advertising, at NeurIPS 2024. This competition focuses on bidding decision-making in uncertain and competitive games. It specifically includes the AIGB (AI-Generated Bidding) track, which emphasizes the innovation of generative algorithms in the field of auto-bidding. This competition has attracted over 1,500 teams from around the world. After six months of intense competition, participants have proposed numerous innovative and forward-looking solutions, significantly promoting research and technological development in this field. In this competition, we have open-sourced AuctionNet, a benchmark for bidding decision-making in large-scale ad auctions derived from a real-world online advertising platform. We have also pre-generated a substantial dataset comprising 10 million ad opportunities, 48 diverse auto-bidding agents, and over 500 million auction records. Performance evaluations of baseline algorithms, such as linear programming, reinforcement learning, and generative models, are also presented as part of AuctionNet. AuctionNet has powered the competition at NeurIPS 2024. We believe that the proposed competition and AuctionNet provide opportunities for more researchers to gain insights and conduct research in this field, driving technical innovation for both academic research and real-world practical applications.'
  volume: 276
  URL: https://proceedings.mlr.press/v276/zhang26a.html
  PDF: https://raw.githubusercontent.com/mlresearch/v276/main/assets/zhang26a/zhang26a.pdf
  edit: https://github.com/mlresearch//v276/edit/gh-pages/_posts/2026-11-21-zhang26a.md
  series: 'Proceedings of Machine Learning Research'
  container-title: 'Proceedings of the NeurIPS 2023-2025 Competition Track'
  publisher: 'PMLR'
  author: 
  - given: Zhilin
    family: Zhang
  - given: Shuai
    family: Dou
  - given: Yusen
    family: Huo
  - given: Zhijian
    family: Duan
  - given: Ningyuan
    family: Li
  - given: Chuan
    family: Yu
  - given: Xiaotie
    family: Deng
  - given: Jian
    family: Xu
  - given: Bo
    family: Zheng
  editor: 
  - given: Tao
    family: Qin
  - given: Kun
    family: Zhang
  - given: Jes
    family: Frellsen
  page: 174-185
  id: zhang26a
  issued:
    date-parts: 
      - 2026
      - 11
      - 21
  firstpage: 174
  lastpage: 185
  published: 2026-11-21 00:00:00 +0000
