Memory Head for Pre-Trained Backbones in Continual Learning

This paper is focused on the role of classification heads for pre-trained backbones in the context of continual learning. A novel neuron model is proposed as basic constituent of what we refer to as Memory Head, which naturally includes self-organized memorization capabilities, going beyond the ones of classic neurons and specifically designed for continual learning purposes. Memory Heads are based on memory units which are indexed depending on the input, with a mechanism which resembles attention models. Such a computational structure allows the head to adapt to different regions of the input space without altering its behavior in other regions, that might be possibly associated to previously acquired knowledge. The neuron model is generic, as it does not exploit any supervisory information and it does not require any experience replay strategies. When stacked on top of pre-trained backbones, the proposed head allows the network to adapt to new knowledge and to memorize the properties of the temporally streamed data. The experimental activity of the paper covers both the case of frozen and fine-tuned backbones, showing that Memory Heads overcome recent state-of-the art competitors which work in the same setting. Moreover, continual online learning is explored in the class-domain incremental setting, being it a more challenging scenario which is less frequently analyzed in the literature. We demonstrate that Memory Heads are more flexible with respect to “vanilla” heads, and more effective than several experience-replay-based approaches.