Extracting Training Data From Document-Based VQA Models

Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, Federico Tombari
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:40813-40826, 2024.

Abstract

Vision-Language Models (VLMs) have made remarkable progress in document-based Visual Question Answering (i.e., responding to queries about the contents of an input document provided as an image). In this work, we show these models can memorize responses for training samples and regurgitate them even when the relevant visual information has been removed. This includes Personal Identifiable Information (PII) repeated once in the training set, indicating these models could divulge memorised sensitive information and therefore pose a privacy risk. We quantitatively measure the extractability of information in controlled experiments and differentiate between cases where it arises from generalization capabilities or from memorization. We further investigate the factors that influence memorization across multiple state-of-the-art models and propose an effective heuristic countermeasure that empirically prevents the extractability of PII.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-pinto24a, title = {Extracting Training Data From Document-Based {VQA} Models}, author = {Pinto, Francesco and Rauschmayr, Nathalie and Tram\`{e}r, Florian and Torr, Philip and Tombari, Federico}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {40813--40826}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/pinto24a/pinto24a.pdf}, url = {https://proceedings.mlr.press/v235/pinto24a.html}, abstract = {Vision-Language Models (VLMs) have made remarkable progress in document-based Visual Question Answering (i.e., responding to queries about the contents of an input document provided as an image). In this work, we show these models can memorize responses for training samples and regurgitate them even when the relevant visual information has been removed. This includes Personal Identifiable Information (PII) repeated once in the training set, indicating these models could divulge memorised sensitive information and therefore pose a privacy risk. We quantitatively measure the extractability of information in controlled experiments and differentiate between cases where it arises from generalization capabilities or from memorization. We further investigate the factors that influence memorization across multiple state-of-the-art models and propose an effective heuristic countermeasure that empirically prevents the extractability of PII.} }
Endnote
%0 Conference Paper %T Extracting Training Data From Document-Based VQA Models %A Francesco Pinto %A Nathalie Rauschmayr %A Florian Tramèr %A Philip Torr %A Federico Tombari %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-pinto24a %I PMLR %P 40813--40826 %U https://proceedings.mlr.press/v235/pinto24a.html %V 235 %X Vision-Language Models (VLMs) have made remarkable progress in document-based Visual Question Answering (i.e., responding to queries about the contents of an input document provided as an image). In this work, we show these models can memorize responses for training samples and regurgitate them even when the relevant visual information has been removed. This includes Personal Identifiable Information (PII) repeated once in the training set, indicating these models could divulge memorised sensitive information and therefore pose a privacy risk. We quantitatively measure the extractability of information in controlled experiments and differentiate between cases where it arises from generalization capabilities or from memorization. We further investigate the factors that influence memorization across multiple state-of-the-art models and propose an effective heuristic countermeasure that empirically prevents the extractability of PII.
APA
Pinto, F., Rauschmayr, N., Tramèr, F., Torr, P. & Tombari, F.. (2024). Extracting Training Data From Document-Based VQA Models. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:40813-40826 Available from https://proceedings.mlr.press/v235/pinto24a.html.

Related Material