Learning with Partial Forgetting in Modern Hopfield Networks

Toshihiro Ota, Ikuro Sato, Rei Kawakami, Masayuki Tanaka, Nakamasa Inoue
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:6661-6673, 2023.

Abstract

It has been known by neuroscience studies that partial and transient forgetting of memory often plays an important role in the brain to improve performance for certain intellectual activities. In machine learning, associative memory models such as classical and modern Hopfield networks have been proposed to express memories as attractors in the feature space of a closed recurrent network. In this work, we propose learning with partial forgetting (LwPF), where a partial forgetting functionality is designed by element-wise non-bijective projections, for memory neurons in modern Hopfield networks to improve model performance. We incorporate LwPF into the attention mechanism also, whose process has been shown to be identical to the update rule of a certain modern Hopfield network, by modifying the corresponding Lagrangian. We evaluated the effectiveness of LwPF on three diverse tasks such as bit-pattern classification, immune repertoire classification for computational biology, and image classification for computer vision, and confirmed that LwPF consistently improves the performance of existing neural networks including DeepRC and vision transformers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-ota23a, title = {Learning with Partial Forgetting in Modern Hopfield Networks}, author = {Ota, Toshihiro and Sato, Ikuro and Kawakami, Rei and Tanaka, Masayuki and Inoue, Nakamasa}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {6661--6673}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/ota23a/ota23a.pdf}, url = {https://proceedings.mlr.press/v206/ota23a.html}, abstract = {It has been known by neuroscience studies that partial and transient forgetting of memory often plays an important role in the brain to improve performance for certain intellectual activities. In machine learning, associative memory models such as classical and modern Hopfield networks have been proposed to express memories as attractors in the feature space of a closed recurrent network. In this work, we propose learning with partial forgetting (LwPF), where a partial forgetting functionality is designed by element-wise non-bijective projections, for memory neurons in modern Hopfield networks to improve model performance. We incorporate LwPF into the attention mechanism also, whose process has been shown to be identical to the update rule of a certain modern Hopfield network, by modifying the corresponding Lagrangian. We evaluated the effectiveness of LwPF on three diverse tasks such as bit-pattern classification, immune repertoire classification for computational biology, and image classification for computer vision, and confirmed that LwPF consistently improves the performance of existing neural networks including DeepRC and vision transformers.} }
Endnote
%0 Conference Paper %T Learning with Partial Forgetting in Modern Hopfield Networks %A Toshihiro Ota %A Ikuro Sato %A Rei Kawakami %A Masayuki Tanaka %A Nakamasa Inoue %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-ota23a %I PMLR %P 6661--6673 %U https://proceedings.mlr.press/v206/ota23a.html %V 206 %X It has been known by neuroscience studies that partial and transient forgetting of memory often plays an important role in the brain to improve performance for certain intellectual activities. In machine learning, associative memory models such as classical and modern Hopfield networks have been proposed to express memories as attractors in the feature space of a closed recurrent network. In this work, we propose learning with partial forgetting (LwPF), where a partial forgetting functionality is designed by element-wise non-bijective projections, for memory neurons in modern Hopfield networks to improve model performance. We incorporate LwPF into the attention mechanism also, whose process has been shown to be identical to the update rule of a certain modern Hopfield network, by modifying the corresponding Lagrangian. We evaluated the effectiveness of LwPF on three diverse tasks such as bit-pattern classification, immune repertoire classification for computational biology, and image classification for computer vision, and confirmed that LwPF consistently improves the performance of existing neural networks including DeepRC and vision transformers.
APA
Ota, T., Sato, I., Kawakami, R., Tanaka, M. & Inoue, N.. (2023). Learning with Partial Forgetting in Modern Hopfield Networks. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:6661-6673 Available from https://proceedings.mlr.press/v206/ota23a.html.

Related Material