Approximate Decentralized Bayesian Inference

This paper presents an approximate method for performing Bayesian inference in models with conditional independence over a decentralized network of learning agents. The method first employs variational inference on each individual learning agent to generate a local approximate posterior, the agents transmit their local poste- riors to other agents in the network, and finally each agent combines its set of received local pos- teriors. The key insight in this work is that, for many Bayesian models, approximate inference schemes destroy symmetry and dependencies in the model that are crucial to the correct appli- cation of Bayes’ rule when combining the lo- cal posteriors. The proposed method addresses this issue by including an additional optimization step in the combination procedure that accounts for these broken dependencies. Experiments on synthetic and real data demonstrate that the de- centralized method provides advantages in com- putational performance and predictive test likeli- hood over previous batch and distributed meth- ods.