Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li, Lianghe Shi, Jinxin Zhou, Zhihui Zhu, Liyue Shen, Qing Qu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:67615-67639, 2026.

Abstract

Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners. However, the connection between these two abilities remains less explored. In this work, drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-li26br, title = {Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles}, author = {Li, Xiao and Jia, Yixuan and Zhang, Zekai and Li, Xiang and Shi, Lianghe and Zhou, Jinxin and Zhu, Zhihui and Shen, Liyue and Qu, Qing}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {67615--67639}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26br/li26br.pdf}, url = {https://proceedings.mlr.press/v306/li26br.html}, abstract = {Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners. However, the connection between these two abilities remains less explored. In this work, drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.} }
Endnote
%0 Conference Paper %T Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles %A Xiao Li %A Yixuan Jia %A Zekai Zhang %A Xiang Li %A Lianghe Shi %A Jinxin Zhou %A Zhihui Zhu %A Liyue Shen %A Qing Qu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26br %I PMLR %P 67615--67639 %U https://proceedings.mlr.press/v306/li26br.html %V 306 %X Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners. However, the connection between these two abilities remains less explored. In this work, drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.
APA
Li, X., Jia, Y., Zhang, Z., Li, X., Shi, L., Zhou, J., Zhu, Z., Shen, L. & Qu, Q.. (2026). Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:67615-67639 Available from https://proceedings.mlr.press/v306/li26br.html.

Related Material