Improved Generalization Bounds for Robust Learning
[edit]
Proceedings of the 30th International Conference on Algorithmic Learning Theory, PMLR 98:162183, 2019.
Abstract
We consider a model of robust learning in an adversarial
environment. The learner gets uncorrupted training data with access
to possible corruptions that may be effected by the adversary during
testing. The learner’s goal is to build a robust classifier that would be
tested on future adversarial examples. We use a zerosum game
between the learner and the adversary as our game theoretic
framework. The adversary is limited to $k$ possible corruptions for
each input. Our model is closely related to the adversarial examples
model of Schmidt et al. (2018); Madry et al. (2017).
Our main results consist of generalization bounds for the binary and
multiclass classification, as well as the realvalued case (regression).
For the binary classification setting, we both tighten the generalization bound of
Feige, Mansour, and Schapire (2015), and also are able to handle an infinite hypothesis class $H$.
The sample complexity is improved from
$O(\frac{1}{\epsilon^4}\log(\frac{H}{\delta}))$ to $O(\frac{1}{\epsilon^2}(k\log(k)VC(H)+\log\frac{1}{\delta}))$.
Additionally, we extend the algorithm and generalization bound from the binary
to the multiclass and realvalued cases. Along the way, we obtain results on fatshattering dimension
and Rademacher complexity of $k$fold maxima over function classes; these may be of independent interest.
For binary classification, the algorithm of Feige et al. (2015) uses a regret minimization algorithm
and an ERM oracle as a blackbox; we adapt it for the multiclass and regression settings.
The algorithm provides us with near optimal policies for the players on a given training sample.
Related Material


