Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization

Hiroshi Sawada, Kazuo Aoyama, Yuya Hikima
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:53063-53079, 2025.

Abstract

This paper proposes a novel concept of natural perturbations for black-box training of neural networks by zeroth-order optimization. When a neural network is implemented directly in hardware, training its parameters by backpropagation ends up with an inaccurate result due to the lack of detailed internal information. We instead employ zeroth-order optimization, where the sampling of parameter perturbations is of great importance. The sampling strategy we propose maximizes the entropy of perturbations with a regularization that the probability distribution conditioned by the neural network does not change drastically, by inheriting the concept of natural gradient. Experimental results show the superiority of our proposal on diverse datasets, tasks, and architectures.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-sawada25a, title = {Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization}, author = {Sawada, Hiroshi and Aoyama, Kazuo and Hikima, Yuya}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {53063--53079}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/sawada25a/sawada25a.pdf}, url = {https://proceedings.mlr.press/v267/sawada25a.html}, abstract = {This paper proposes a novel concept of natural perturbations for black-box training of neural networks by zeroth-order optimization. When a neural network is implemented directly in hardware, training its parameters by backpropagation ends up with an inaccurate result due to the lack of detailed internal information. We instead employ zeroth-order optimization, where the sampling of parameter perturbations is of great importance. The sampling strategy we propose maximizes the entropy of perturbations with a regularization that the probability distribution conditioned by the neural network does not change drastically, by inheriting the concept of natural gradient. Experimental results show the superiority of our proposal on diverse datasets, tasks, and architectures.} }
Endnote
%0 Conference Paper %T Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization %A Hiroshi Sawada %A Kazuo Aoyama %A Yuya Hikima %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-sawada25a %I PMLR %P 53063--53079 %U https://proceedings.mlr.press/v267/sawada25a.html %V 267 %X This paper proposes a novel concept of natural perturbations for black-box training of neural networks by zeroth-order optimization. When a neural network is implemented directly in hardware, training its parameters by backpropagation ends up with an inaccurate result due to the lack of detailed internal information. We instead employ zeroth-order optimization, where the sampling of parameter perturbations is of great importance. The sampling strategy we propose maximizes the entropy of perturbations with a regularization that the probability distribution conditioned by the neural network does not change drastically, by inheriting the concept of natural gradient. Experimental results show the superiority of our proposal on diverse datasets, tasks, and architectures.
APA
Sawada, H., Aoyama, K. & Hikima, Y.. (2025). Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:53063-53079 Available from https://proceedings.mlr.press/v267/sawada25a.html.

Related Material