Determinantal Point Processes for Mini-Batch Diversification

Cheng Zhang, Hedvig Kjellstrom, Stephan Mandt
Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, PMLR R15:281-290, 2017.

Abstract

We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classi- cal SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a sim- ilarity measure between data points and gives low probabilities to mini-batches which contain redun- dant data, and higher probabilities to mini-batches with more diverse data. This simultaneously bal- ances the data and leads to stochastic gradients with lower variance. We term this approach Di- versified Mini-Batch SGD (DM-SGD). We show that regular SGD and a biased version of stratified sampling emerge as special cases. Furthermore, DM-SGD generalizes stratified sampling to cases where no discrete features exist to bin the data into groups. We show experimentally that our method results more interpretable and diverse features in unsupervised setups, and in better classification accuracies in supervised setups.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR15-zhang17a, title = {Determinantal Point Processes for Mini-Batch Diversification}, author = {Zhang, Cheng and Kjellstrom, Hedvig and Mandt, Stephan}, booktitle = {Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence}, pages = {281--290}, year = {2017}, editor = {Elidan, Gal and Kersting, Kristian}, volume = {R15}, series = {Proceedings of Machine Learning Research}, month = {11--15 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r15/main/assets/zhang17a/zhang17a.pdf}, url = {https://proceedings.mlr.press/r15/zhang17a.html}, abstract = {We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classi- cal SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a sim- ilarity measure between data points and gives low probabilities to mini-batches which contain redun- dant data, and higher probabilities to mini-batches with more diverse data. This simultaneously bal- ances the data and leads to stochastic gradients with lower variance. We term this approach Di- versified Mini-Batch SGD (DM-SGD). We show that regular SGD and a biased version of stratified sampling emerge as special cases. Furthermore, DM-SGD generalizes stratified sampling to cases where no discrete features exist to bin the data into groups. We show experimentally that our method results more interpretable and diverse features in unsupervised setups, and in better classification accuracies in supervised setups.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Determinantal Point Processes for Mini-Batch Diversification %A Cheng Zhang %A Hedvig Kjellstrom %A Stephan Mandt %B Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2017 %E Gal Elidan %E Kristian Kersting %F pmlr-vR15-zhang17a %I PMLR %P 281--290 %U https://proceedings.mlr.press/r15/zhang17a.html %V R15 %X We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classi- cal SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a sim- ilarity measure between data points and gives low probabilities to mini-batches which contain redun- dant data, and higher probabilities to mini-batches with more diverse data. This simultaneously bal- ances the data and leads to stochastic gradients with lower variance. We term this approach Di- versified Mini-Batch SGD (DM-SGD). We show that regular SGD and a biased version of stratified sampling emerge as special cases. Furthermore, DM-SGD generalizes stratified sampling to cases where no discrete features exist to bin the data into groups. We show experimentally that our method results more interpretable and diverse features in unsupervised setups, and in better classification accuracies in supervised setups. %Z Reissued by PMLR on 04 October 2026.
APA
Zhang, C., Kjellstrom, H. & Mandt, S.. (2017). Determinantal Point Processes for Mini-Batch Diversification. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R15:281-290 Available from https://proceedings.mlr.press/r15/zhang17a.html. Reissued by PMLR on 04 October 2026.

Related Material