A Bayesian Information-Theoretic Approach to Data Attribution

Dharmesh Tailor, Nicolò Felicioni, Kamil Ciosek
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:1936-1944, 2026.

Abstract

Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the information loss they induce—the entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-tailor26a, title = { A Bayesian Information-Theoretic Approach to Data Attribution }, author = {Tailor, Dharmesh and Felicioni, Nicol\`{o} and Ciosek, Kamil}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {1936--1944}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/tailor26a/tailor26a.pdf}, url = {https://proceedings.mlr.press/v300/tailor26a.html}, abstract = { Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the information loss they induce—the entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice. } }
Endnote
%0 Conference Paper %T A Bayesian Information-Theoretic Approach to Data Attribution %A Dharmesh Tailor %A Nicolò Felicioni %A Kamil Ciosek %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-tailor26a %I PMLR %P 1936--1944 %U https://proceedings.mlr.press/v300/tailor26a.html %V 300 %X Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the information loss they induce—the entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice.
APA
Tailor, D., Felicioni, N. & Ciosek, K.. (2026). A Bayesian Information-Theoretic Approach to Data Attribution . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:1936-1944 Available from https://proceedings.mlr.press/v300/tailor26a.html.

Related Material