Position: Why We Must Rethink Empirical Research in Machine Learning

Moritz Herrmann, F. Julian D. Lange, Katharina Eggensperger, Giuseppe Casalicchio, Marcel Wever, Matthias Feurer, David Rügamer, Eyke Hüllermeier, Anne-Laure Boulesteix, Bernd Bischl
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:18228-18247, 2024.

Abstract

We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-herrmann24b, title = {Position: Why We Must Rethink Empirical Research in Machine Learning}, author = {Herrmann, Moritz and Lange, F. Julian D. and Eggensperger, Katharina and Casalicchio, Giuseppe and Wever, Marcel and Feurer, Matthias and R\"{u}gamer, David and H\"{u}llermeier, Eyke and Boulesteix, Anne-Laure and Bischl, Bernd}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {18228--18247}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/herrmann24b/herrmann24b.pdf}, url = {https://proceedings.mlr.press/v235/herrmann24b.html}, abstract = {We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.} }
Endnote
%0 Conference Paper %T Position: Why We Must Rethink Empirical Research in Machine Learning %A Moritz Herrmann %A F. Julian D. Lange %A Katharina Eggensperger %A Giuseppe Casalicchio %A Marcel Wever %A Matthias Feurer %A David Rügamer %A Eyke Hüllermeier %A Anne-Laure Boulesteix %A Bernd Bischl %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-herrmann24b %I PMLR %P 18228--18247 %U https://proceedings.mlr.press/v235/herrmann24b.html %V 235 %X We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.
APA
Herrmann, M., Lange, F.J.D., Eggensperger, K., Casalicchio, G., Wever, M., Feurer, M., Rügamer, D., Hüllermeier, E., Boulesteix, A. & Bischl, B.. (2024). Position: Why We Must Rethink Empirical Research in Machine Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:18228-18247 Available from https://proceedings.mlr.press/v235/herrmann24b.html.

Related Material