Alternating Direction Method of Multipliers for Quantization

Tianjian Huang, Prajwal Singhania, Maziar Sanjabi, Pabitra Mitra, Meisam Razaviyayn
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:208-216, 2021.

Abstract

Quantization of the parameters of machine learning models, such as deep neural networks, requires solving constrained optimization problems, where the constraint set is formed by the Cartesian product of many simple discrete sets. For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization (ADMM-Q) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem. We establish the convergence of the iterates of ADMM-Q to certain stationary points. To the best of our knowledge, this is the first analysis of an ADMM-type method for problems with discrete variables/constraints. Based on our theoretical insights, we develop a few variants of ADMM-Q that can handle inexact update rules, and have improved performance via the use of "soft projection" and "injecting randomness to the algorithm". We empirically evaluate the efficacy of our proposed approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-huang21a, title = { Alternating Direction Method of Multipliers for Quantization }, author = {Huang, Tianjian and Singhania, Prajwal and Sanjabi, Maziar and Mitra, Pabitra and Razaviyayn, Meisam}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {208--216}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/huang21a/huang21a.pdf}, url = {https://proceedings.mlr.press/v130/huang21a.html}, abstract = { Quantization of the parameters of machine learning models, such as deep neural networks, requires solving constrained optimization problems, where the constraint set is formed by the Cartesian product of many simple discrete sets. For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization (ADMM-Q) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem. We establish the convergence of the iterates of ADMM-Q to certain stationary points. To the best of our knowledge, this is the first analysis of an ADMM-type method for problems with discrete variables/constraints. Based on our theoretical insights, we develop a few variants of ADMM-Q that can handle inexact update rules, and have improved performance via the use of "soft projection" and "injecting randomness to the algorithm". We empirically evaluate the efficacy of our proposed approaches. } }
Endnote
%0 Conference Paper %T Alternating Direction Method of Multipliers for Quantization %A Tianjian Huang %A Prajwal Singhania %A Maziar Sanjabi %A Pabitra Mitra %A Meisam Razaviyayn %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-huang21a %I PMLR %P 208--216 %U https://proceedings.mlr.press/v130/huang21a.html %V 130 %X Quantization of the parameters of machine learning models, such as deep neural networks, requires solving constrained optimization problems, where the constraint set is formed by the Cartesian product of many simple discrete sets. For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization (ADMM-Q) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem. We establish the convergence of the iterates of ADMM-Q to certain stationary points. To the best of our knowledge, this is the first analysis of an ADMM-type method for problems with discrete variables/constraints. Based on our theoretical insights, we develop a few variants of ADMM-Q that can handle inexact update rules, and have improved performance via the use of "soft projection" and "injecting randomness to the algorithm". We empirically evaluate the efficacy of our proposed approaches.
APA
Huang, T., Singhania, P., Sanjabi, M., Mitra, P. & Razaviyayn, M.. (2021). Alternating Direction Method of Multipliers for Quantization . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:208-216 Available from https://proceedings.mlr.press/v130/huang21a.html.

Related Material