Modern Hopfield Networks meet Encoded Neural Representations - Addressing Practical Considerations

Satyananda Kashyap, Niharika S. D’Souza, Luyao Shi, Ken C. L. Wong, Hongzhi Wang, Tanveer Syeda-mahmood
Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, PMLR 285:115-127, 2024.

Abstract

Content-addressable memories such as Modern Hopfield Networks (MHN) have been studied as mathematical models of the auto-association and storage/retrieval in the human declarative memory, yet their practical use for large-scale content storage faces challenges. Chief among them is the occurrence of meta-stable states, particularly when handling large amounts of high dimensional content. This paper introduces Hopfield Encoding Networks (HEN), a framework that integrates encoded neural representations into MHNs to improve pattern separability and reduce meta-stable states. We show that HEN can also be used for retrieval in the context of hetero association of images with natural language queries, thus removing the limitation of requiring access to partial content in the same domain. Experimental results demonstrate substantial reduction in meta-stable states and increased storage capacity while still enabling perfect recall of a significantly larger number of inputs advancing the practical utility of associative memory networks for real-world tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v285-kashyap24a, title = {Modern Hopfield Networks meet Encoded Neural Representations - Addressing Practical Considerations}, author = {Kashyap, Satyananda and D'Souza, Niharika S. and Shi, Luyao and Wong, Ken C. L. and Wang, Hongzhi and Syeda-mahmood, Tanveer}, booktitle = {Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models}, pages = {115--127}, year = {2024}, editor = {Fumero, Marco and Domine, Clementine and Lähner, Zorah and Crisostomi, Donato and Moschella, Luca and Stachenfeld, Kimberly}, volume = {285}, series = {Proceedings of Machine Learning Research}, month = {14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v285/main/assets/kashyap24a/kashyap24a.pdf}, url = {https://proceedings.mlr.press/v285/kashyap24a.html}, abstract = {Content-addressable memories such as Modern Hopfield Networks (MHN) have been studied as mathematical models of the auto-association and storage/retrieval in the human declarative memory, yet their practical use for large-scale content storage faces challenges. Chief among them is the occurrence of meta-stable states, particularly when handling large amounts of high dimensional content. This paper introduces Hopfield Encoding Networks (HEN), a framework that integrates encoded neural representations into MHNs to improve pattern separability and reduce meta-stable states. We show that HEN can also be used for retrieval in the context of hetero association of images with natural language queries, thus removing the limitation of requiring access to partial content in the same domain. Experimental results demonstrate substantial reduction in meta-stable states and increased storage capacity while still enabling perfect recall of a significantly larger number of inputs advancing the practical utility of associative memory networks for real-world tasks.} }
Endnote
%0 Conference Paper %T Modern Hopfield Networks meet Encoded Neural Representations - Addressing Practical Considerations %A Satyananda Kashyap %A Niharika S. D’Souza %A Luyao Shi %A Ken C. L. Wong %A Hongzhi Wang %A Tanveer Syeda-mahmood %B Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models %C Proceedings of Machine Learning Research %D 2024 %E Marco Fumero %E Clementine Domine %E Zorah Lähner %E Donato Crisostomi %E Luca Moschella %E Kimberly Stachenfeld %F pmlr-v285-kashyap24a %I PMLR %P 115--127 %U https://proceedings.mlr.press/v285/kashyap24a.html %V 285 %X Content-addressable memories such as Modern Hopfield Networks (MHN) have been studied as mathematical models of the auto-association and storage/retrieval in the human declarative memory, yet their practical use for large-scale content storage faces challenges. Chief among them is the occurrence of meta-stable states, particularly when handling large amounts of high dimensional content. This paper introduces Hopfield Encoding Networks (HEN), a framework that integrates encoded neural representations into MHNs to improve pattern separability and reduce meta-stable states. We show that HEN can also be used for retrieval in the context of hetero association of images with natural language queries, thus removing the limitation of requiring access to partial content in the same domain. Experimental results demonstrate substantial reduction in meta-stable states and increased storage capacity while still enabling perfect recall of a significantly larger number of inputs advancing the practical utility of associative memory networks for real-world tasks.
APA
Kashyap, S., D’Souza, N.S., Shi, L., Wong, K.C.L., Wang, H. & Syeda-mahmood, T.. (2024). Modern Hopfield Networks meet Encoded Neural Representations - Addressing Practical Considerations. Proceedings of UniReps: the Second Edition of the Workshop on Unifying Representations in Neural Models, in Proceedings of Machine Learning Research 285:115-127 Available from https://proceedings.mlr.press/v285/kashyap24a.html.

Related Material