PAC-Net: A Model Pruning Approach to Inductive Transfer Learning

Sanghoon Myung, In Huh, Wonik Jang, Jae Myung Choe, Jisu Ryu, Daesin Kim, Kee-Eung Kim, Changwook Jeong
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:16240-16252, 2022.

Abstract

Inductive transfer learning aims to learn from a small amount of training data for the target task by utilizing a pre-trained model from the source task. Most strategies that involve large-scale deep learning models adopt initialization with the pre-trained model and fine-tuning for the target task. However, when using over-parameterized models, we can often prune the model without sacrificing the accuracy of the source task. This motivates us to adopt model pruning for transfer learning with deep learning models. In this paper, we propose PAC-Net, a simple yet effective approach for transfer learning based on pruning. PAC-Net consists of three steps: Prune, Allocate, and Calibrate (PAC). The main idea behind these steps is to identify essential weights for the source task, fine-tune on the source task by updating the essential weights, and then calibrate on the target task by updating the remaining redundant weights. Under the various and extensive set of inductive transfer learning experiments, we show that our method achieves state-of-the-art performance by a large margin.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-myung22a, title = {{PAC}-Net: A Model Pruning Approach to Inductive Transfer Learning}, author = {Myung, Sanghoon and Huh, In and Jang, Wonik and Choe, Jae Myung and Ryu, Jisu and Kim, Daesin and Kim, Kee-Eung and Jeong, Changwook}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {16240--16252}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/myung22a/myung22a.pdf}, url = {https://proceedings.mlr.press/v162/myung22a.html}, abstract = {Inductive transfer learning aims to learn from a small amount of training data for the target task by utilizing a pre-trained model from the source task. Most strategies that involve large-scale deep learning models adopt initialization with the pre-trained model and fine-tuning for the target task. However, when using over-parameterized models, we can often prune the model without sacrificing the accuracy of the source task. This motivates us to adopt model pruning for transfer learning with deep learning models. In this paper, we propose PAC-Net, a simple yet effective approach for transfer learning based on pruning. PAC-Net consists of three steps: Prune, Allocate, and Calibrate (PAC). The main idea behind these steps is to identify essential weights for the source task, fine-tune on the source task by updating the essential weights, and then calibrate on the target task by updating the remaining redundant weights. Under the various and extensive set of inductive transfer learning experiments, we show that our method achieves state-of-the-art performance by a large margin.} }
Endnote
%0 Conference Paper %T PAC-Net: A Model Pruning Approach to Inductive Transfer Learning %A Sanghoon Myung %A In Huh %A Wonik Jang %A Jae Myung Choe %A Jisu Ryu %A Daesin Kim %A Kee-Eung Kim %A Changwook Jeong %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-myung22a %I PMLR %P 16240--16252 %U https://proceedings.mlr.press/v162/myung22a.html %V 162 %X Inductive transfer learning aims to learn from a small amount of training data for the target task by utilizing a pre-trained model from the source task. Most strategies that involve large-scale deep learning models adopt initialization with the pre-trained model and fine-tuning for the target task. However, when using over-parameterized models, we can often prune the model without sacrificing the accuracy of the source task. This motivates us to adopt model pruning for transfer learning with deep learning models. In this paper, we propose PAC-Net, a simple yet effective approach for transfer learning based on pruning. PAC-Net consists of three steps: Prune, Allocate, and Calibrate (PAC). The main idea behind these steps is to identify essential weights for the source task, fine-tune on the source task by updating the essential weights, and then calibrate on the target task by updating the remaining redundant weights. Under the various and extensive set of inductive transfer learning experiments, we show that our method achieves state-of-the-art performance by a large margin.
APA
Myung, S., Huh, I., Jang, W., Choe, J.M., Ryu, J., Kim, D., Kim, K. & Jeong, C.. (2022). PAC-Net: A Model Pruning Approach to Inductive Transfer Learning. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:16240-16252 Available from https://proceedings.mlr.press/v162/myung22a.html.

Related Material