Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not

Chelsea Murray, James U. Allingham, Javier Antorán, José Miguel Hernández-Lobato
Proceedings on "I (Still) Can't Believe It's Not Better!" at NeurIPS 2021 Workshops, PMLR 163:59-63, 2022.

Abstract

Farquhar et al. [2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance. For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance. They suggest that this is due to an “overfitting bias” which offsets the active learning bias. We show that depth uncertainty networks operate in a low overfitting regime, much like underparameterised models. They should therefore see an increase in performance with bias correction. Surprisingly, they do not. We propose that this negative result, as well as the results Farquhar et al. [2021], can be explained via the lens of the bias-variance decomposition of generalisation error.

Cite this Paper

BibTeX
@InProceedings{pmlr-v163-murray22a, title = {Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not}, author = {Murray, Chelsea and Allingham, James U. and Antor\'{a}n, Javier and Hern\'{a}ndez-Lobato, Jos\'{e} Miguel}, booktitle = {Proceedings on "I (Still) Can't Believe It's Not Better!" at NeurIPS 2021 Workshops}, pages = {59--63}, year = {2022}, editor = {Pradier, Melanie F. and Schein, Aaron and Hyland, Stephanie and Ruiz, Francisco J. R. and Forde, Jessica Z.}, volume = {163}, series = {Proceedings of Machine Learning Research}, month = {13 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v163/murray22a/murray22a.pdf}, url = {https://proceedings.mlr.press/v163/murray22a.html}, abstract = {Farquhar et al. [2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance. For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance. They suggest that this is due to an “overfitting bias” which offsets the active learning bias. We show that depth uncertainty networks operate in a low overfitting regime, much like underparameterised models. They should therefore see an increase in performance with bias correction. Surprisingly, they do not. We propose that this negative result, as well as the results Farquhar et al. [2021], can be explained via the lens of the bias-variance decomposition of generalisation error.} }
Endnote
%0 Conference Paper %T Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not %A Chelsea Murray %A James U. Allingham %A Javier Antorán %A José Miguel Hernández-Lobato %B Proceedings on "I (Still) Can't Believe It's Not Better!" at NeurIPS 2021 Workshops %C Proceedings of Machine Learning Research %D 2022 %E Melanie F. Pradier %E Aaron Schein %E Stephanie Hyland %E Francisco J. R. Ruiz %E Jessica Z. Forde %F pmlr-v163-murray22a %I PMLR %P 59--63 %U https://proceedings.mlr.press/v163/murray22a.html %V 163 %X Farquhar et al. [2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance. For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance. They suggest that this is due to an “overfitting bias” which offsets the active learning bias. We show that depth uncertainty networks operate in a low overfitting regime, much like underparameterised models. They should therefore see an increase in performance with bias correction. Surprisingly, they do not. We propose that this negative result, as well as the results Farquhar et al. [2021], can be explained via the lens of the bias-variance decomposition of generalisation error.
APA
Murray, C., Allingham, J.U., Antorán, J. & Hernández-Lobato, J.M.. (2022). Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not. Proceedings on "I (Still) Can't Believe It's Not Better!" at NeurIPS 2021 Workshops, in Proceedings of Machine Learning Research 163:59-63 Available from https://proceedings.mlr.press/v163/murray22a.html.

Related Material