Convex Bounds on the Softmax Function with Applications to Robustness Verification

Dennis Wei, Haoze Wu, Min Wu, Pin-Yu Chen, Clark Barrett, Eitan Farchi
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:6853-6878, 2023.

Abstract

The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-wei23c, title = {Convex Bounds on the Softmax Function with Applications to Robustness Verification}, author = {Wei, Dennis and Wu, Haoze and Wu, Min and Chen, Pin-Yu and Barrett, Clark and Farchi, Eitan}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {6853--6878}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/wei23c/wei23c.pdf}, url = {https://proceedings.mlr.press/v206/wei23c.html}, abstract = {The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.} }
Endnote
%0 Conference Paper %T Convex Bounds on the Softmax Function with Applications to Robustness Verification %A Dennis Wei %A Haoze Wu %A Min Wu %A Pin-Yu Chen %A Clark Barrett %A Eitan Farchi %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-wei23c %I PMLR %P 6853--6878 %U https://proceedings.mlr.press/v206/wei23c.html %V 206 %X The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.
APA
Wei, D., Wu, H., Wu, M., Chen, P., Barrett, C. & Farchi, E.. (2023). Convex Bounds on the Softmax Function with Applications to Robustness Verification. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:6853-6878 Available from https://proceedings.mlr.press/v206/wei23c.html.

Related Material