Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation

Sayedmoslem Shokrolahi, Jae-Mo Kang, Il-Min Kim
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:112713-112741, 2026.

Abstract

Text-to-image (T2I) diffusion models (DMs) have achieved remarkable generative quality but still exhibit risk of producing biased and inappropriate images. A promising line of prior work aims to mitigate this issue by learning interpretable, linearly controllable concepts from semantic spaces, such as the U-Net bottleneck; however, these methods rely entirely on U-Net architectures and cannot be generalized to modern ViT-based DMs, including FLUX and PixArt. In this work, we present an architecture-agnostic framework for discovering interpretable and linearly controllable semantic attributes across any T2I DM backbone. Theoretically, we show that multi-modal attention heads in ViT-based DMs exhibit a linear semantic structure: injected concept vectors combine linearly at the attention level and induce near-linear effects at the model output, satisfying homogeneity and additivity. These theoretical results are aligned with and supported by empirical experiments. Building on this insight, we introduce a method that learns external concept vectors, which are added to the multi-modal attention heads for ViT-based DMs or to the bottleneck layer for U-Net-based DMs, while keeping pretrained models frozen. Experiments across SDXL, SD3.5, PixArt, and FLUX demonstrate that these concept vectors provide interpretability, linearity, and significantly improved fairness while preserving visual fidelity. The code and demo are available at https://github.com/Moslem-Sh21/responsible-t2i-diffusion.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-shokrolahi26a, title = {Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation}, author = {Shokrolahi, Sayedmoslem and Kang, Jae-Mo and Kim, Il-Min}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {112713--112741}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/shokrolahi26a/shokrolahi26a.pdf}, url = {https://proceedings.mlr.press/v306/shokrolahi26a.html}, abstract = {Text-to-image (T2I) diffusion models (DMs) have achieved remarkable generative quality but still exhibit risk of producing biased and inappropriate images. A promising line of prior work aims to mitigate this issue by learning interpretable, linearly controllable concepts from semantic spaces, such as the U-Net bottleneck; however, these methods rely entirely on U-Net architectures and cannot be generalized to modern ViT-based DMs, including FLUX and PixArt. In this work, we present an architecture-agnostic framework for discovering interpretable and linearly controllable semantic attributes across any T2I DM backbone. Theoretically, we show that multi-modal attention heads in ViT-based DMs exhibit a linear semantic structure: injected concept vectors combine linearly at the attention level and induce near-linear effects at the model output, satisfying homogeneity and additivity. These theoretical results are aligned with and supported by empirical experiments. Building on this insight, we introduce a method that learns external concept vectors, which are added to the multi-modal attention heads for ViT-based DMs or to the bottleneck layer for U-Net-based DMs, while keeping pretrained models frozen. Experiments across SDXL, SD3.5, PixArt, and FLUX demonstrate that these concept vectors provide interpretability, linearity, and significantly improved fairness while preserving visual fidelity. The code and demo are available at https://github.com/Moslem-Sh21/responsible-t2i-diffusion.} }
Endnote
%0 Conference Paper %T Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation %A Sayedmoslem Shokrolahi %A Jae-Mo Kang %A Il-Min Kim %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-shokrolahi26a %I PMLR %P 112713--112741 %U https://proceedings.mlr.press/v306/shokrolahi26a.html %V 306 %X Text-to-image (T2I) diffusion models (DMs) have achieved remarkable generative quality but still exhibit risk of producing biased and inappropriate images. A promising line of prior work aims to mitigate this issue by learning interpretable, linearly controllable concepts from semantic spaces, such as the U-Net bottleneck; however, these methods rely entirely on U-Net architectures and cannot be generalized to modern ViT-based DMs, including FLUX and PixArt. In this work, we present an architecture-agnostic framework for discovering interpretable and linearly controllable semantic attributes across any T2I DM backbone. Theoretically, we show that multi-modal attention heads in ViT-based DMs exhibit a linear semantic structure: injected concept vectors combine linearly at the attention level and induce near-linear effects at the model output, satisfying homogeneity and additivity. These theoretical results are aligned with and supported by empirical experiments. Building on this insight, we introduce a method that learns external concept vectors, which are added to the multi-modal attention heads for ViT-based DMs or to the bottleneck layer for U-Net-based DMs, while keeping pretrained models frozen. Experiments across SDXL, SD3.5, PixArt, and FLUX demonstrate that these concept vectors provide interpretability, linearity, and significantly improved fairness while preserving visual fidelity. The code and demo are available at https://github.com/Moslem-Sh21/responsible-t2i-diffusion.
APA
Shokrolahi, S., Kang, J. & Kim, I.. (2026). Responsible Text-to-Image Diffusion: Interpretable and Linearly Controllable Semantics for Fair and Safe Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:112713-112741 Available from https://proceedings.mlr.press/v306/shokrolahi26a.html.

Related Material