On the Misinformation in a Statistical Experiment

Jake Callahan, Tommie Catanach
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2287-2295, 2026.

Abstract

The principle that more informative experiments are always better is a cornerstone of Bayesian experimental design. This principle assumes the practitioner’s model and inference are correct. In practice, both the data-generating model and the inferential approximation are inevitably misspecified, and we show that under these conditions the classical framework for comparing experiments breaks down. Designs ranked as most informative can become actively harmful, amplifying bias to produce confident but incorrect inferences. We demonstrate that the commonly-accepted axioms of experimental utility, such as Blackwell monotonicity, fail under misspecification, and that information measures proposed to handle it, like the Expected Generalized Information Gain (EGIG), do not obey these axioms. To resolve this, we propose a generalized axiomatic framework for robust Bayesian experimental design. We prove that EGIG satisfies our axioms as a criterion that penalizes inferential error, providing a principled foundation for its use in Bayesian experimental design. As a complementary approach, we derive a new measure that instead penalizes model error. Finally, we demonstrate our framework’s utility across common modes of misspecification, showing it provides a reliable guide for experimental design where classical methods fail.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-callahan26a, title = { On the Misinformation in a Statistical Experiment }, author = {Callahan, Jake and Catanach, Tommie}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2287--2295}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/callahan26a/callahan26a.pdf}, url = {https://proceedings.mlr.press/v300/callahan26a.html}, abstract = { The principle that more informative experiments are always better is a cornerstone of Bayesian experimental design. This principle assumes the practitioner’s model and inference are correct. In practice, both the data-generating model and the inferential approximation are inevitably misspecified, and we show that under these conditions the classical framework for comparing experiments breaks down. Designs ranked as most informative can become actively harmful, amplifying bias to produce confident but incorrect inferences. We demonstrate that the commonly-accepted axioms of experimental utility, such as Blackwell monotonicity, fail under misspecification, and that information measures proposed to handle it, like the Expected Generalized Information Gain (EGIG), do not obey these axioms. To resolve this, we propose a generalized axiomatic framework for robust Bayesian experimental design. We prove that EGIG satisfies our axioms as a criterion that penalizes inferential error, providing a principled foundation for its use in Bayesian experimental design. As a complementary approach, we derive a new measure that instead penalizes model error. Finally, we demonstrate our framework’s utility across common modes of misspecification, showing it provides a reliable guide for experimental design where classical methods fail. } }
Endnote
%0 Conference Paper %T On the Misinformation in a Statistical Experiment %A Jake Callahan %A Tommie Catanach %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-callahan26a %I PMLR %P 2287--2295 %U https://proceedings.mlr.press/v300/callahan26a.html %V 300 %X The principle that more informative experiments are always better is a cornerstone of Bayesian experimental design. This principle assumes the practitioner’s model and inference are correct. In practice, both the data-generating model and the inferential approximation are inevitably misspecified, and we show that under these conditions the classical framework for comparing experiments breaks down. Designs ranked as most informative can become actively harmful, amplifying bias to produce confident but incorrect inferences. We demonstrate that the commonly-accepted axioms of experimental utility, such as Blackwell monotonicity, fail under misspecification, and that information measures proposed to handle it, like the Expected Generalized Information Gain (EGIG), do not obey these axioms. To resolve this, we propose a generalized axiomatic framework for robust Bayesian experimental design. We prove that EGIG satisfies our axioms as a criterion that penalizes inferential error, providing a principled foundation for its use in Bayesian experimental design. As a complementary approach, we derive a new measure that instead penalizes model error. Finally, we demonstrate our framework’s utility across common modes of misspecification, showing it provides a reliable guide for experimental design where classical methods fail.
APA
Callahan, J. & Catanach, T.. (2026). On the Misinformation in a Statistical Experiment . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2287-2295 Available from https://proceedings.mlr.press/v300/callahan26a.html.

Related Material