Proactive Detection of Voice Cloning with Localized Watermarking

Robin San Roman, Pierre Fernandez, Hady Elsahar, Alexandre Défossez, Teddy Furon, Tuan Tran
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:43180-43196, 2024.

Abstract

In the rapidly evolving field of speech generative models, there is a pressing need to ensure audio authenticity against the risks of voice cloning. We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator / detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level, and a novel perceptual loss inspired by auditory masking, that enables AudioSeal to achieve better imperceptibility. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics. Additionally, AudioSeal is designed with a fast, single-pass detector, that significantly surpasses existing models in speed, achieving detection up to two orders of magnitude faster, making it ideal for large-scale and real-time applications.Code is available at https://github.com/facebookresearch/audioseal

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-san-roman24a, title = {Proactive Detection of Voice Cloning with Localized Watermarking}, author = {San Roman, Robin and Fernandez, Pierre and Elsahar, Hady and D\'{e}fossez, Alexandre and Furon, Teddy and Tran, Tuan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {43180--43196}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/san-roman24a/san-roman24a.pdf}, url = {https://proceedings.mlr.press/v235/san-roman24a.html}, abstract = {In the rapidly evolving field of speech generative models, there is a pressing need to ensure audio authenticity against the risks of voice cloning. We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator / detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level, and a novel perceptual loss inspired by auditory masking, that enables AudioSeal to achieve better imperceptibility. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics. Additionally, AudioSeal is designed with a fast, single-pass detector, that significantly surpasses existing models in speed, achieving detection up to two orders of magnitude faster, making it ideal for large-scale and real-time applications.Code is available at https://github.com/facebookresearch/audioseal} }
Endnote
%0 Conference Paper %T Proactive Detection of Voice Cloning with Localized Watermarking %A Robin San Roman %A Pierre Fernandez %A Hady Elsahar %A Alexandre Défossez %A Teddy Furon %A Tuan Tran %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-san-roman24a %I PMLR %P 43180--43196 %U https://proceedings.mlr.press/v235/san-roman24a.html %V 235 %X In the rapidly evolving field of speech generative models, there is a pressing need to ensure audio authenticity against the risks of voice cloning. We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator / detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level, and a novel perceptual loss inspired by auditory masking, that enables AudioSeal to achieve better imperceptibility. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics. Additionally, AudioSeal is designed with a fast, single-pass detector, that significantly surpasses existing models in speed, achieving detection up to two orders of magnitude faster, making it ideal for large-scale and real-time applications.Code is available at https://github.com/facebookresearch/audioseal
APA
San Roman, R., Fernandez, P., Elsahar, H., Défossez, A., Furon, T. & Tran, T.. (2024). Proactive Detection of Voice Cloning with Localized Watermarking. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:43180-43196 Available from https://proceedings.mlr.press/v235/san-roman24a.html.

Related Material