VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference

Sakshi Agarwal, Gabriel Hope, Jimin Heo, Erik B. Sudderth
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:4240-4248, 2026.

Abstract

Diffusion probabilistic models learn to remove noise added during training, generating novel data (e.g., images) from Gaussian noise through sequential denoising. However, conditioning the generative process on corrupted or masked images is challenging. While various methods have been proposed for inpainting masked images with diffusion priors, they often fail to produce samples from the true conditional distribution, especially for large masked regions. Additionally, many can’t be applied to latent diffusion models which have been demonstrated to generate high-quality images at a significantly lower computational cost. We propose a hierarchical variational inference algorithm that optimizes a non-Gaussian Markov approximation of the true diffusion posterior. Our VIPaint method outperforms existing approaches to inpainting, producing diverse high-quality imputations, while also being effective for other inverse problems like deblurring and superresolution.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-agarwal26a, title = { VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference }, author = {Agarwal, Sakshi and Hope, Gabriel and Heo, Jimin and Sudderth, Erik B.}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {4240--4248}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/agarwal26a/agarwal26a.pdf}, url = {https://proceedings.mlr.press/v300/agarwal26a.html}, abstract = { Diffusion probabilistic models learn to remove noise added during training, generating novel data (e.g., images) from Gaussian noise through sequential denoising. However, conditioning the generative process on corrupted or masked images is challenging. While various methods have been proposed for inpainting masked images with diffusion priors, they often fail to produce samples from the true conditional distribution, especially for large masked regions. Additionally, many can’t be applied to latent diffusion models which have been demonstrated to generate high-quality images at a significantly lower computational cost. We propose a hierarchical variational inference algorithm that optimizes a non-Gaussian Markov approximation of the true diffusion posterior. Our VIPaint method outperforms existing approaches to inpainting, producing diverse high-quality imputations, while also being effective for other inverse problems like deblurring and superresolution. } }
Endnote
%0 Conference Paper %T VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference %A Sakshi Agarwal %A Gabriel Hope %A Jimin Heo %A Erik B. Sudderth %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-agarwal26a %I PMLR %P 4240--4248 %U https://proceedings.mlr.press/v300/agarwal26a.html %V 300 %X Diffusion probabilistic models learn to remove noise added during training, generating novel data (e.g., images) from Gaussian noise through sequential denoising. However, conditioning the generative process on corrupted or masked images is challenging. While various methods have been proposed for inpainting masked images with diffusion priors, they often fail to produce samples from the true conditional distribution, especially for large masked regions. Additionally, many can’t be applied to latent diffusion models which have been demonstrated to generate high-quality images at a significantly lower computational cost. We propose a hierarchical variational inference algorithm that optimizes a non-Gaussian Markov approximation of the true diffusion posterior. Our VIPaint method outperforms existing approaches to inpainting, producing diverse high-quality imputations, while also being effective for other inverse problems like deblurring and superresolution.
APA
Agarwal, S., Hope, G., Heo, J. & Sudderth, E.B.. (2026). VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:4240-4248 Available from https://proceedings.mlr.press/v300/agarwal26a.html.

Related Material