Prediction-Oriented Bayesian Active Learning

Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, Tom Rainforth
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:7331-7348, 2023.

Abstract

Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-bickfordsmith23a, title = {Prediction-Oriented Bayesian Active Learning}, author = {Bickford Smith, Freddie and Kirsch, Andreas and Farquhar, Sebastian and Gal, Yarin and Foster, Adam and Rainforth, Tom}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {7331--7348}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/bickfordsmith23a/bickfordsmith23a.pdf}, url = {https://proceedings.mlr.press/v206/bickfordsmith23a.html}, abstract = {Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.} }
Endnote
%0 Conference Paper %T Prediction-Oriented Bayesian Active Learning %A Freddie Bickford Smith %A Andreas Kirsch %A Sebastian Farquhar %A Yarin Gal %A Adam Foster %A Tom Rainforth %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-bickfordsmith23a %I PMLR %P 7331--7348 %U https://proceedings.mlr.press/v206/bickfordsmith23a.html %V 206 %X Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.
APA
Bickford Smith, F., Kirsch, A., Farquhar, S., Gal, Y., Foster, A. & Rainforth, T.. (2023). Prediction-Oriented Bayesian Active Learning. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:7331-7348 Available from https://proceedings.mlr.press/v206/bickfordsmith23a.html.

Related Material