Sample as you Infer: Predictive Coding with Langevin Dynamics

Umais Zahid, Qinghai Guo, Zafeirios Fountas
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:58105-58121, 2024.

Abstract

We present Langevin Predictive Coding (LPC), a novel algorithm for deep generative model learning that builds upon the predictive coding framework of computational neuroscience. By injecting Gaussian noise into the predictive coding inference procedure and incorporating an encoder network initialization, we reframe the approach as an amortized Langevin sampling method for optimizing a tight variational lower bound. To increase robustness to sampling step size, we present a lightweight preconditioning technique inspired by Riemannian Langevin methods and adaptive SGD. We compare LPC against VAEs by training generative models on benchmark datasets; our experiments demonstrate superior sample quality and faster convergence for LPC in a fraction of SGD training iterations, while matching or exceeding VAE performance across key metrics like FID, diversity and coverage.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zahid24a, title = {Sample as you Infer: Predictive Coding with {L}angevin Dynamics}, author = {Zahid, Umais and Guo, Qinghai and Fountas, Zafeirios}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {58105--58121}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zahid24a/zahid24a.pdf}, url = {https://proceedings.mlr.press/v235/zahid24a.html}, abstract = {We present Langevin Predictive Coding (LPC), a novel algorithm for deep generative model learning that builds upon the predictive coding framework of computational neuroscience. By injecting Gaussian noise into the predictive coding inference procedure and incorporating an encoder network initialization, we reframe the approach as an amortized Langevin sampling method for optimizing a tight variational lower bound. To increase robustness to sampling step size, we present a lightweight preconditioning technique inspired by Riemannian Langevin methods and adaptive SGD. We compare LPC against VAEs by training generative models on benchmark datasets; our experiments demonstrate superior sample quality and faster convergence for LPC in a fraction of SGD training iterations, while matching or exceeding VAE performance across key metrics like FID, diversity and coverage.} }
Endnote
%0 Conference Paper %T Sample as you Infer: Predictive Coding with Langevin Dynamics %A Umais Zahid %A Qinghai Guo %A Zafeirios Fountas %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zahid24a %I PMLR %P 58105--58121 %U https://proceedings.mlr.press/v235/zahid24a.html %V 235 %X We present Langevin Predictive Coding (LPC), a novel algorithm for deep generative model learning that builds upon the predictive coding framework of computational neuroscience. By injecting Gaussian noise into the predictive coding inference procedure and incorporating an encoder network initialization, we reframe the approach as an amortized Langevin sampling method for optimizing a tight variational lower bound. To increase robustness to sampling step size, we present a lightweight preconditioning technique inspired by Riemannian Langevin methods and adaptive SGD. We compare LPC against VAEs by training generative models on benchmark datasets; our experiments demonstrate superior sample quality and faster convergence for LPC in a fraction of SGD training iterations, while matching or exceeding VAE performance across key metrics like FID, diversity and coverage.
APA
Zahid, U., Guo, Q. & Fountas, Z.. (2024). Sample as you Infer: Predictive Coding with Langevin Dynamics. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:58105-58121 Available from https://proceedings.mlr.press/v235/zahid24a.html.

Related Material