Explicit Inductive Bias for Transfer Learning with Convolutional Networks

Xuhong LI, Yves Grandvalet, Franck Davoine
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:2825-2834, 2018.

Abstract

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-li18a, title = {Explicit Inductive Bias for Transfer Learning with Convolutional Networks}, author = {LI, Xuhong and Grandvalet, Yves and Davoine, Franck}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {2825--2834}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/li18a/li18a.pdf}, url = {https://proceedings.mlr.press/v80/li18a.html}, abstract = {In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.} }
Endnote
%0 Conference Paper %T Explicit Inductive Bias for Transfer Learning with Convolutional Networks %A Xuhong LI %A Yves Grandvalet %A Franck Davoine %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-li18a %I PMLR %P 2825--2834 %U https://proceedings.mlr.press/v80/li18a.html %V 80 %X In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.
APA
LI, X., Grandvalet, Y. & Davoine, F.. (2018). Explicit Inductive Bias for Transfer Learning with Convolutional Networks. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:2825-2834 Available from https://proceedings.mlr.press/v80/li18a.html.

Related Material