Evaluating the Adversarial Robustness of Adaptive Test-time Defenses

Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, Taylan Cemgil
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4421-4435, 2022.

Abstract

Adaptive defenses, which optimize at test time, promise to improve adversarial robustness. We categorize such adaptive test-time defenses, explain their potential benefits and drawbacks, and evaluate a representative variety of the latest adaptive defenses for image classification. Unfortunately, none significantly improve upon static defenses when subjected to our careful case study evaluation. Some even weaken the underlying static model while simultaneously increasing inference computation. While these results are disappointing, we still believe that adaptive test-time defenses are a promising avenue of research and, as such, we provide recommendations for their thorough evaluation. We extend the checklist of Carlini et al. (2019) by providing concrete steps specific to adaptive defenses.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-croce22a, title = {Evaluating the Adversarial Robustness of Adaptive Test-time Defenses}, author = {Croce, Francesco and Gowal, Sven and Brunner, Thomas and Shelhamer, Evan and Hein, Matthias and Cemgil, Taylan}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {4421--4435}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/croce22a/croce22a.pdf}, url = {https://proceedings.mlr.press/v162/croce22a.html}, abstract = {Adaptive defenses, which optimize at test time, promise to improve adversarial robustness. We categorize such adaptive test-time defenses, explain their potential benefits and drawbacks, and evaluate a representative variety of the latest adaptive defenses for image classification. Unfortunately, none significantly improve upon static defenses when subjected to our careful case study evaluation. Some even weaken the underlying static model while simultaneously increasing inference computation. While these results are disappointing, we still believe that adaptive test-time defenses are a promising avenue of research and, as such, we provide recommendations for their thorough evaluation. We extend the checklist of Carlini et al. (2019) by providing concrete steps specific to adaptive defenses.} }
Endnote
%0 Conference Paper %T Evaluating the Adversarial Robustness of Adaptive Test-time Defenses %A Francesco Croce %A Sven Gowal %A Thomas Brunner %A Evan Shelhamer %A Matthias Hein %A Taylan Cemgil %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-croce22a %I PMLR %P 4421--4435 %U https://proceedings.mlr.press/v162/croce22a.html %V 162 %X Adaptive defenses, which optimize at test time, promise to improve adversarial robustness. We categorize such adaptive test-time defenses, explain their potential benefits and drawbacks, and evaluate a representative variety of the latest adaptive defenses for image classification. Unfortunately, none significantly improve upon static defenses when subjected to our careful case study evaluation. Some even weaken the underlying static model while simultaneously increasing inference computation. While these results are disappointing, we still believe that adaptive test-time defenses are a promising avenue of research and, as such, we provide recommendations for their thorough evaluation. We extend the checklist of Carlini et al. (2019) by providing concrete steps specific to adaptive defenses.
APA
Croce, F., Gowal, S., Brunner, T., Shelhamer, E., Hein, M. & Cemgil, T.. (2022). Evaluating the Adversarial Robustness of Adaptive Test-time Defenses. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:4421-4435 Available from https://proceedings.mlr.press/v162/croce22a.html.

Related Material