Pitfalls in Machine Learning Research: Reexamining the Development Cycle

Stella Biderman, Walter J. Scheirer
Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, PMLR 137:106-117, 2020.

Abstract

Applied machine learning research has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the applied machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.

Cite this Paper

BibTeX
@InProceedings{pmlr-v137-biderman20a, title = {Pitfalls in Machine Learning Research: Reexamining the Development Cycle}, author = {Biderman, Stella and Scheirer, Walter J.}, booktitle = {Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops}, pages = {106--117}, year = {2020}, editor = {Zosa Forde, Jessica and Ruiz, Francisco and Pradier, Melanie F. and Schein, Aaron}, volume = {137}, series = {Proceedings of Machine Learning Research}, month = {12 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v137/biderman20a/biderman20a.pdf}, url = {https://proceedings.mlr.press/v137/biderman20a.html}, abstract = {Applied machine learning research has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the applied machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.} }
Endnote
%0 Conference Paper %T Pitfalls in Machine Learning Research: Reexamining the Development Cycle %A Stella Biderman %A Walter J. Scheirer %B Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops %C Proceedings of Machine Learning Research %D 2020 %E Jessica Zosa Forde %E Francisco Ruiz %E Melanie F. Pradier %E Aaron Schein %F pmlr-v137-biderman20a %I PMLR %P 106--117 %U https://proceedings.mlr.press/v137/biderman20a.html %V 137 %X Applied machine learning research has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the applied machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.
APA
Biderman, S. & Scheirer, W.J.. (2020). Pitfalls in Machine Learning Research: Reexamining the Development Cycle. Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, in Proceedings of Machine Learning Research 137:106-117 Available from https://proceedings.mlr.press/v137/biderman20a.html.

Related Material