Alethia: a Foundational Encoder for Voice Deepfakes

Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, Surya Koppisetti
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:166939-166961, 2026.

Abstract

Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhu26l, title = {Alethia: a Foundational Encoder for Voice Deepfakes}, author = {Zhu, Yi and Dwivedi, Brahmi and Raghuram, Jayaram and Koppisetti, Surya}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {166939--166961}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhu26l/zhu26l.pdf}, url = {https://proceedings.mlr.press/v306/zhu26l.html}, abstract = {Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.} }
Endnote
%0 Conference Paper %T Alethia: a Foundational Encoder for Voice Deepfakes %A Yi Zhu %A Brahmi Dwivedi %A Jayaram Raghuram %A Surya Koppisetti %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhu26l %I PMLR %P 166939--166961 %U https://proceedings.mlr.press/v306/zhu26l.html %V 306 %X Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.
APA
Zhu, Y., Dwivedi, B., Raghuram, J. & Koppisetti, S.. (2026). Alethia: a Foundational Encoder for Voice Deepfakes. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:166939-166961 Available from https://proceedings.mlr.press/v306/zhu26l.html.

Related Material