ByzantineRobust Distributed Learning: Towards Optimal Statistical Rates
[edit]
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:56505659, 2018.
Abstract
In this paper, we develop distributed optimization algorithms that are provably robust against Byzantine failures—arbitrary and potentially adversarial behavior, in distributed computing systems, with a focus on achieving optimal statistical performance. A main result of this work is a sharp analysis of two robust distributed gradient descent algorithms based on median and trimmed mean operations, respectively. We prove statistical error rates for all of strongly convex, nonstrongly convex, and smooth nonconvex population loss functions. In particular, these algorithms are shown to achieve orderoptimal statistical error rates for strongly convex losses. To achieve better communication efficiency, we further propose a medianbased distributed algorithm that is provably robust, and uses only one communication round. For strongly convex quadratic loss, we show that this algorithm achieves the same optimal error rate as the robust distributed gradient descent algorithms.
Related Material


