Benchmarking Non-Invasive Speech BCIs: Lessons Learned from the 2025 PNPL Competition

Gereon Elvers, Gilad Landau, Francesco Mantegna, Miran Özdogan, Tasha Kim, Teyun Kwon, SungJun Cho, Benjamin Ballyk, Luisa Kurth, Dulhan Jayalath, Pratik Somaiya, Chetan Gohil, Brendan Shillingford, Greg Farquhar, Minqi Jiang, Caglar Gulcehre, Xabier de Zuazo, Hamza Abdelhedi, Yorguin Mantilla Ramos, Karim Jerbi, Mark Woolrich, Natalie Voets, Oiwi Parker Jones
Proceedings of the NeurIPS 2023-2025 Competition Track, PMLR 276:69-89, 2026.

Abstract

We present a retrospective on the 2025 PNPL Competition, an open machine-learning benchmark for decoding speech from non-invasive brain recordings, which ran from June to September 2025 and culminated in a workshop at NeurIPS that December. The competition used the LibriBrain dataset — over 50 hours of within-subject MEG with standard data splits — and challenged participants to tackle two tasks: Speech Detection (binary frame-level classification) and Phoneme Classification (39-class ARPAbet prediction). Across 155 registered teams and 6,041 submissions, top F1-macro scores reached 95.6% for Speech Detection and 73.6% for Phoneme Classification, exceeding prior baselines of 68% and 44%, respectively, by wide margins. High-performing submissions converged on a common architectural template — temporal CNN or TCN backbones with task-specific recurrent or attention-based heads — whilst task-specific strategies such as temporal smoothing and ensemble voting proved critical for competitive performance. Hidden track evaluations suggest that main-track Speech Detection models generalised poorly to balanced class distributions. This could imply a reliance on class priors rather than robust neural decoding, highlighting a useful lesson: evaluation distributions should reflect intended deployment conditions. We further reflect on design choices, organisational lessons, and plans for future competitions in this series.

Cite this Paper


BibTeX
@InProceedings{pmlr-v276-elvers26a, title = {Benchmarking Non-Invasive Speech {BCIs}: Lessons Learned from the 2025 {PNPL} Competition}, author = {Elvers, Gereon and Landau, Gilad and Mantegna, Francesco and {\"O}zdogan, Miran and Kim, Tasha and Kwon, Teyun and Cho, SungJun and Ballyk, Benjamin and Kurth, Luisa and Jayalath, Dulhan and Somaiya, Pratik and Gohil, Chetan and Shillingford, Brendan and Farquhar, Greg and Jiang, Minqi and Gulcehre, Caglar and de Zuazo, Xabier and Abdelhedi, Hamza and Mantilla Ramos, Yorguin and Jerbi, Karim and Woolrich, Mark and Voets, Natalie and Parker Jones, Oiwi}, booktitle = {Proceedings of the NeurIPS 2023-2025 Competition Track}, pages = {69--89}, year = {2026}, editor = {Qin, Tao and Zhang, Kun and Frellsen, Jes}, volume = {276}, series = {Proceedings of Machine Learning Research}, month = {02--07 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v276/main/assets/elvers26a/elvers26a.pdf}, url = {https://proceedings.mlr.press/v276/elvers26a.html}, abstract = {We present a retrospective on the 2025 PNPL Competition, an open machine-learning benchmark for decoding speech from non-invasive brain recordings, which ran from June to September 2025 and culminated in a workshop at NeurIPS that December. The competition used the LibriBrain dataset — over 50 hours of within-subject MEG with standard data splits — and challenged participants to tackle two tasks: Speech Detection (binary frame-level classification) and Phoneme Classification (39-class ARPAbet prediction). Across 155 registered teams and 6,041 submissions, top F1-macro scores reached 95.6% for Speech Detection and 73.6% for Phoneme Classification, exceeding prior baselines of 68% and 44%, respectively, by wide margins. High-performing submissions converged on a common architectural template — temporal CNN or TCN backbones with task-specific recurrent or attention-based heads — whilst task-specific strategies such as temporal smoothing and ensemble voting proved critical for competitive performance. Hidden track evaluations suggest that main-track Speech Detection models generalised poorly to balanced class distributions. This could imply a reliance on class priors rather than robust neural decoding, highlighting a useful lesson: evaluation distributions should reflect intended deployment conditions. We further reflect on design choices, organisational lessons, and plans for future competitions in this series.} }
Endnote
%0 Conference Paper %T Benchmarking Non-Invasive Speech BCIs: Lessons Learned from the 2025 PNPL Competition %A Gereon Elvers %A Gilad Landau %A Francesco Mantegna %A Miran Özdogan %A Tasha Kim %A Teyun Kwon %A SungJun Cho %A Benjamin Ballyk %A Luisa Kurth %A Dulhan Jayalath %A Pratik Somaiya %A Chetan Gohil %A Brendan Shillingford %A Greg Farquhar %A Minqi Jiang %A Caglar Gulcehre %A Xabier de Zuazo %A Hamza Abdelhedi %A Yorguin Mantilla Ramos %A Karim Jerbi %A Mark Woolrich %A Natalie Voets %A Oiwi Parker Jones %B Proceedings of the NeurIPS 2023-2025 Competition Track %C Proceedings of Machine Learning Research %D 2026 %E Tao Qin %E Kun Zhang %E Jes Frellsen %F pmlr-v276-elvers26a %I PMLR %P 69--89 %U https://proceedings.mlr.press/v276/elvers26a.html %V 276 %X We present a retrospective on the 2025 PNPL Competition, an open machine-learning benchmark for decoding speech from non-invasive brain recordings, which ran from June to September 2025 and culminated in a workshop at NeurIPS that December. The competition used the LibriBrain dataset — over 50 hours of within-subject MEG with standard data splits — and challenged participants to tackle two tasks: Speech Detection (binary frame-level classification) and Phoneme Classification (39-class ARPAbet prediction). Across 155 registered teams and 6,041 submissions, top F1-macro scores reached 95.6% for Speech Detection and 73.6% for Phoneme Classification, exceeding prior baselines of 68% and 44%, respectively, by wide margins. High-performing submissions converged on a common architectural template — temporal CNN or TCN backbones with task-specific recurrent or attention-based heads — whilst task-specific strategies such as temporal smoothing and ensemble voting proved critical for competitive performance. Hidden track evaluations suggest that main-track Speech Detection models generalised poorly to balanced class distributions. This could imply a reliance on class priors rather than robust neural decoding, highlighting a useful lesson: evaluation distributions should reflect intended deployment conditions. We further reflect on design choices, organisational lessons, and plans for future competitions in this series.
APA
Elvers, G., Landau, G., Mantegna, F., Özdogan, M., Kim, T., Kwon, T., Cho, S., Ballyk, B., Kurth, L., Jayalath, D., Somaiya, P., Gohil, C., Shillingford, B., Farquhar, G., Jiang, M., Gulcehre, C., de Zuazo, X., Abdelhedi, H., Mantilla Ramos, Y., Jerbi, K., Woolrich, M., Voets, N. & Parker Jones, O.. (2026). Benchmarking Non-Invasive Speech BCIs: Lessons Learned from the 2025 PNPL Competition. Proceedings of the NeurIPS 2023-2025 Competition Track, in Proceedings of Machine Learning Research 276:69-89 Available from https://proceedings.mlr.press/v276/elvers26a.html.

Related Material