Sequential and Parallel Constrained Max-value Entropy Search via Information Lower Bound

Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, Masayuki Karasuyama
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:20960-20986, 2022.

Abstract

Max-value entropy search (MES) is one of the state-of-the-art approaches in Bayesian optimization (BO). In this paper, we propose a novel variant of MES for constrained problems, called Constrained MES via Information lower BOund (CMES-IBO), that is based on a Monte Carlo (MC) estimator of a lower bound of a mutual information (MI). Unlike existing studies, our MI is defined so that uncertainty with respect to feasibility can be incorporated. We derive a lower bound of the MI that guarantees non-negativity, while a constrained counterpart of conventional MES can be negative. We further provide theoretical analysis that assures the low-variability of our estimator which has never been investigated for any existing information-theoretic BO. Moreover, using the conditional MI, we extend CMES-IBO to the parallel setting while maintaining the desirable properties. We demonstrate the effectiveness of CMES-IBO by several benchmark functions and real-world problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-takeno22a, title = {Sequential and Parallel Constrained Max-value Entropy Search via Information Lower Bound}, author = {Takeno, Shion and Tamura, Tomoyuki and Shitara, Kazuki and Karasuyama, Masayuki}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {20960--20986}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/takeno22a/takeno22a.pdf}, url = {https://proceedings.mlr.press/v162/takeno22a.html}, abstract = {Max-value entropy search (MES) is one of the state-of-the-art approaches in Bayesian optimization (BO). In this paper, we propose a novel variant of MES for constrained problems, called Constrained MES via Information lower BOund (CMES-IBO), that is based on a Monte Carlo (MC) estimator of a lower bound of a mutual information (MI). Unlike existing studies, our MI is defined so that uncertainty with respect to feasibility can be incorporated. We derive a lower bound of the MI that guarantees non-negativity, while a constrained counterpart of conventional MES can be negative. We further provide theoretical analysis that assures the low-variability of our estimator which has never been investigated for any existing information-theoretic BO. Moreover, using the conditional MI, we extend CMES-IBO to the parallel setting while maintaining the desirable properties. We demonstrate the effectiveness of CMES-IBO by several benchmark functions and real-world problems.} }
Endnote
%0 Conference Paper %T Sequential and Parallel Constrained Max-value Entropy Search via Information Lower Bound %A Shion Takeno %A Tomoyuki Tamura %A Kazuki Shitara %A Masayuki Karasuyama %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-takeno22a %I PMLR %P 20960--20986 %U https://proceedings.mlr.press/v162/takeno22a.html %V 162 %X Max-value entropy search (MES) is one of the state-of-the-art approaches in Bayesian optimization (BO). In this paper, we propose a novel variant of MES for constrained problems, called Constrained MES via Information lower BOund (CMES-IBO), that is based on a Monte Carlo (MC) estimator of a lower bound of a mutual information (MI). Unlike existing studies, our MI is defined so that uncertainty with respect to feasibility can be incorporated. We derive a lower bound of the MI that guarantees non-negativity, while a constrained counterpart of conventional MES can be negative. We further provide theoretical analysis that assures the low-variability of our estimator which has never been investigated for any existing information-theoretic BO. Moreover, using the conditional MI, we extend CMES-IBO to the parallel setting while maintaining the desirable properties. We demonstrate the effectiveness of CMES-IBO by several benchmark functions and real-world problems.
APA
Takeno, S., Tamura, T., Shitara, K. & Karasuyama, M.. (2022). Sequential and Parallel Constrained Max-value Entropy Search via Information Lower Bound. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:20960-20986 Available from https://proceedings.mlr.press/v162/takeno22a.html.

Related Material