Gradient-Free Structured Pruning with Unlabeled Data

Azade Nova, Hanjun Dai, Dale Schuurmans
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:26326-26341, 2023.

Abstract

Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customize these models, the need to provide efficient inference has increased. Many efforts have attempted to reduce inference cost through model compression techniques such as pruning and distillation. However, these techniques either require labeled data, or are time-consuming as they require the compressed model to be retrained to regain accuracy. In this paper, we propose a gradient-free structured pruning framework that uses only unlabeled data. An evaluation on the GLUE and SQuAD benchmarks using BERT$_{BASE}$ and DistilBERT illustrates the effectiveness of the proposed approach. By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a $4%$ accuracy loss across all tasks considered.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-nova23a, title = {Gradient-Free Structured Pruning with Unlabeled Data}, author = {Nova, Azade and Dai, Hanjun and Schuurmans, Dale}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {26326--26341}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/nova23a/nova23a.pdf}, url = {https://proceedings.mlr.press/v202/nova23a.html}, abstract = {Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customize these models, the need to provide efficient inference has increased. Many efforts have attempted to reduce inference cost through model compression techniques such as pruning and distillation. However, these techniques either require labeled data, or are time-consuming as they require the compressed model to be retrained to regain accuracy. In this paper, we propose a gradient-free structured pruning framework that uses only unlabeled data. An evaluation on the GLUE and SQuAD benchmarks using BERT$_{BASE}$ and DistilBERT illustrates the effectiveness of the proposed approach. By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a $4%$ accuracy loss across all tasks considered.} }
Endnote
%0 Conference Paper %T Gradient-Free Structured Pruning with Unlabeled Data %A Azade Nova %A Hanjun Dai %A Dale Schuurmans %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-nova23a %I PMLR %P 26326--26341 %U https://proceedings.mlr.press/v202/nova23a.html %V 202 %X Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customize these models, the need to provide efficient inference has increased. Many efforts have attempted to reduce inference cost through model compression techniques such as pruning and distillation. However, these techniques either require labeled data, or are time-consuming as they require the compressed model to be retrained to regain accuracy. In this paper, we propose a gradient-free structured pruning framework that uses only unlabeled data. An evaluation on the GLUE and SQuAD benchmarks using BERT$_{BASE}$ and DistilBERT illustrates the effectiveness of the proposed approach. By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a $4%$ accuracy loss across all tasks considered.
APA
Nova, A., Dai, H. & Schuurmans, D.. (2023). Gradient-Free Structured Pruning with Unlabeled Data. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:26326-26341 Available from https://proceedings.mlr.press/v202/nova23a.html.

Related Material