Likelihood hacking in probabilistic program synthesis

Jacek Karwowski, Younesse Kaddar, Zihuiwen Ye, Esmeralda S. Whitammer, Sam Staton
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:2744-2791, 2026.

Abstract

When language models are trained by reinforcement learning ({RL}) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment $L_{safe}$ satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement $L_{safe}$’s conditions as SafeStan, a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated {Bayesian} model discovery.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-karwowski26a, title = {Likelihood hacking in probabilistic program synthesis}, author = {Karwowski, Jacek and Kaddar, Younesse and Ye, Zihuiwen and Whitammer, Esmeralda S. and Staton, Sam}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {2744--2791}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/karwowski26a/karwowski26a.pdf}, url = {https://proceedings.mlr.press/v337/karwowski26a.html}, abstract = {When language models are trained by reinforcement learning ({RL}) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment $L_{safe}$ satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement $L_{safe}$’s conditions as SafeStan, a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated {Bayesian} model discovery.} }
Endnote
%0 Conference Paper %T Likelihood hacking in probabilistic program synthesis %A Jacek Karwowski %A Younesse Kaddar %A Zihuiwen Ye %A Esmeralda S. Whitammer %A Sam Staton %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-karwowski26a %I PMLR %P 2744--2791 %U https://proceedings.mlr.press/v337/karwowski26a.html %V 337 %X When language models are trained by reinforcement learning ({RL}) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment $L_{safe}$ satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement $L_{safe}$’s conditions as SafeStan, a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated {Bayesian} model discovery.
APA
Karwowski, J., Kaddar, Y., Ye, Z., Whitammer, E.S. & Staton, S.. (2026). Likelihood hacking in probabilistic program synthesis. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:2744-2791 Available from https://proceedings.mlr.press/v337/karwowski26a.html.

Related Material