Minimal Achievable Sufficient Statistic Learning
[edit]
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:14651474, 2019.
Abstract
We introduce Minimal Achievable Sufficient Statistic (MASS) Learning, a machine learning training objective for which the minima are minimal sufficient statistics with respect to a class of functions being optimized over (e.g., deep networks). In deriving MASS Learning, we also introduce Conserved Differential Information (CDI), an informationtheoretic quantity that {—} unlike standard mutual information {—} can be usefully applied to deterministicallydependent continuous random variables like the input and output of a deep network. In a series of experiments, we show that deep networks trained with MASS Learning achieve competitive performance on supervised learning, regularization, and uncertainty quantification benchmarks.
Related Material


