Deep Classifier Mimicry without Data Access

Steven Braun, Martin Mundt, Kristian Kersting
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:4762-4770, 2024.

Abstract

Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model’s decision boundary. We empirically corroborate CAKE’s effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-braun24b, title = {Deep Classifier Mimicry without Data Access}, author = {Braun, Steven and Mundt, Martin and Kersting, Kristian}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {4762--4770}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/braun24b/braun24b.pdf}, url = {https://proceedings.mlr.press/v238/braun24b.html}, abstract = {Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model’s decision boundary. We empirically corroborate CAKE’s effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.} }
Endnote
%0 Conference Paper %T Deep Classifier Mimicry without Data Access %A Steven Braun %A Martin Mundt %A Kristian Kersting %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-braun24b %I PMLR %P 4762--4770 %U https://proceedings.mlr.press/v238/braun24b.html %V 238 %X Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model’s decision boundary. We empirically corroborate CAKE’s effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.
APA
Braun, S., Mundt, M. & Kersting, K.. (2024). Deep Classifier Mimicry without Data Access. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:4762-4770 Available from https://proceedings.mlr.press/v238/braun24b.html.

Related Material