Beware of Data Leakage from Protein LLM Pretraining

Leon Hermann, Tobias Fiedler, Hoang An Nguyen, Melania Nowicka, Jakub M Bartoszewicz
Proceedings of the 19th Machine Learning in Computational Biology meeting, PMLR 261:106-116, 2024.

Abstract

Pretrained protein language models are becoming increasingly popular as a backbone for protein property inference tasks such as structure prediction or function annotation, accelerating biological research. However, related research oftentimes does not consider the effects of data leakage from pretraining on the actual downstream task, resulting in potentially unrealistic performance estimates. Reported generalization might not necessarily be reproducible for proteins highly dissimilar from the pretraining set. In this work, we measure the effects of data leakage from protein language model pretraining in the domain of protein thermostability prediction. Specifically, we compare two different dataset split strategies: a pretraining-aware split, designed to avoid similarity between pretraining data and the held-out test sets, and a commonly-used naive split, relying on clustering the training data for a downstream task without taking the pretraining data into account. Our experiments suggest that data leakage from language model pretraining shows consistent effects on melting point prediction across all experiments, distorting the measured performance by an average 11.1% compared to the pretraining-aware split. The source code and our dataset splits are available at https://gitlab.com/dacs-hpi/pretraining-aware-hotprot.

Cite this Paper

BibTeX
@InProceedings{pmlr-v261-hermann24a, title = {Beware of Data Leakage from Protein LLM Pretraining}, author = {Hermann, Leon and Fiedler, Tobias and Nguyen, Hoang An and Nowicka, Melania and Bartoszewicz, Jakub M}, booktitle = {Proceedings of the 19th Machine Learning in Computational Biology meeting}, pages = {106--116}, year = {2024}, editor = {Knowles, David A and Mostafavi, Sara}, volume = {261}, series = {Proceedings of Machine Learning Research}, month = {05--06 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v261/main/assets/hermann24a/hermann24a.pdf}, url = {https://proceedings.mlr.press/v261/hermann24a.html}, abstract = {Pretrained protein language models are becoming increasingly popular as a backbone for protein property inference tasks such as structure prediction or function annotation, accelerating biological research. However, related research oftentimes does not consider the effects of data leakage from pretraining on the actual downstream task, resulting in potentially unrealistic performance estimates. Reported generalization might not necessarily be reproducible for proteins highly dissimilar from the pretraining set. In this work, we measure the effects of data leakage from protein language model pretraining in the domain of protein thermostability prediction. Specifically, we compare two different dataset split strategies: a pretraining-aware split, designed to avoid similarity between pretraining data and the held-out test sets, and a commonly-used naive split, relying on clustering the training data for a downstream task without taking the pretraining data into account. Our experiments suggest that data leakage from language model pretraining shows consistent effects on melting point prediction across all experiments, distorting the measured performance by an average 11.1% compared to the pretraining-aware split. The source code and our dataset splits are available at https://gitlab.com/dacs-hpi/pretraining-aware-hotprot.} }
Endnote
%0 Conference Paper %T Beware of Data Leakage from Protein LLM Pretraining %A Leon Hermann %A Tobias Fiedler %A Hoang An Nguyen %A Melania Nowicka %A Jakub M Bartoszewicz %B Proceedings of the 19th Machine Learning in Computational Biology meeting %C Proceedings of Machine Learning Research %D 2024 %E David A Knowles %E Sara Mostafavi %F pmlr-v261-hermann24a %I PMLR %P 106--116 %U https://proceedings.mlr.press/v261/hermann24a.html %V 261 %X Pretrained protein language models are becoming increasingly popular as a backbone for protein property inference tasks such as structure prediction or function annotation, accelerating biological research. However, related research oftentimes does not consider the effects of data leakage from pretraining on the actual downstream task, resulting in potentially unrealistic performance estimates. Reported generalization might not necessarily be reproducible for proteins highly dissimilar from the pretraining set. In this work, we measure the effects of data leakage from protein language model pretraining in the domain of protein thermostability prediction. Specifically, we compare two different dataset split strategies: a pretraining-aware split, designed to avoid similarity between pretraining data and the held-out test sets, and a commonly-used naive split, relying on clustering the training data for a downstream task without taking the pretraining data into account. Our experiments suggest that data leakage from language model pretraining shows consistent effects on melting point prediction across all experiments, distorting the measured performance by an average 11.1% compared to the pretraining-aware split. The source code and our dataset splits are available at https://gitlab.com/dacs-hpi/pretraining-aware-hotprot.
APA
Hermann, L., Fiedler, T., Nguyen, H.A., Nowicka, M. & Bartoszewicz, J.M.. (2024). Beware of Data Leakage from Protein LLM Pretraining. Proceedings of the 19th Machine Learning in Computational Biology meeting, in Proceedings of Machine Learning Research 261:106-116 Available from https://proceedings.mlr.press/v261/hermann24a.html.

Related Material