Assessing Local Generalization Capability in Deep Models

Huan Wang, Nitish Shirish Keskar, Caiming Xiong, Richard Socher
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:2077-2087, 2020.

Abstract

While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima, which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order “smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of a model, as well as an algorithm that optimizes the perturbed model accordingly.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-wang20f, title = {Assessing Local Generalization Capability in Deep Models}, author = {Wang, Huan and Keskar, Nitish Shirish and Xiong, Caiming and Socher, Richard}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {2077--2087}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/wang20f/wang20f.pdf}, url = {https://proceedings.mlr.press/v108/wang20f.html}, abstract = {While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima, which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order “smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of a model, as well as an algorithm that optimizes the perturbed model accordingly. } }
Endnote
%0 Conference Paper %T Assessing Local Generalization Capability in Deep Models %A Huan Wang %A Nitish Shirish Keskar %A Caiming Xiong %A Richard Socher %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-wang20f %I PMLR %P 2077--2087 %U https://proceedings.mlr.press/v108/wang20f.html %V 108 %X While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima, which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order “smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of a model, as well as an algorithm that optimizes the perturbed model accordingly.
APA
Wang, H., Keskar, N.S., Xiong, C. & Socher, R.. (2020). Assessing Local Generalization Capability in Deep Models. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:2077-2087 Available from https://proceedings.mlr.press/v108/wang20f.html.

Related Material