ModelDiff: A Framework for Comparing Learning Algorithms

Harshay Shah, Sung Min Park, Andrew Ilyas, Aleksander Madry
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:30646-30688, 2023.

Abstract

We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-shah23a, title = {{M}odel{D}iff: A Framework for Comparing Learning Algorithms}, author = {Shah, Harshay and Park, Sung Min and Ilyas, Andrew and Madry, Aleksander}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {30646--30688}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/shah23a/shah23a.pdf}, url = {https://proceedings.mlr.press/v202/shah23a.html}, abstract = {We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters.} }
Endnote
%0 Conference Paper %T ModelDiff: A Framework for Comparing Learning Algorithms %A Harshay Shah %A Sung Min Park %A Andrew Ilyas %A Aleksander Madry %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-shah23a %I PMLR %P 30646--30688 %U https://proceedings.mlr.press/v202/shah23a.html %V 202 %X We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters.
APA
Shah, H., Park, S.M., Ilyas, A. & Madry, A.. (2023). ModelDiff: A Framework for Comparing Learning Algorithms. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:30646-30688 Available from https://proceedings.mlr.press/v202/shah23a.html.

Related Material