Robust textual embedding against word-level adversarial attacks

Yichen Yang, Xiaosen Wang, Kun He
Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, PMLR 180:2214-2224, 2022.

Abstract

We attribute the vulnerability of natural language processing models to the fact that similar inputs are converted to dissimilar representations in the embedding space, leading to inconsistent outputs, and we propose a novel robust training method, termed \textit{Fast Triplet Metric Learning (FTML)}. Specifically, we argue that the original sample should have similar representation with its adversarial counterparts and distinguish its representation from other samples for better robustness. To this end, we adopt the triplet metric learning into the standard training to pull words closer to their positive samples (\textit{i.e.}, synonyms) and push away their negative samples (\textit{i.e.}, non-synonyms) in the embedding space. Extensive experiments demonstrate that FTML can significantly promote the model robustness against various advanced adversarial attacks while keeping competitive classification accuracy on original samples. Besides, our method is efficient as it only needs to adjust the embedding and introduces very little overhead on the standard training. Our work shows great potential of improving the textual robustness through robust word embedding.

Cite this Paper

BibTeX
@InProceedings{pmlr-v180-yang22c, title = {Robust textual embedding against word-level adversarial attacks}, author = {Yang, Yichen and Wang, Xiaosen and He, Kun}, booktitle = {Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence}, pages = {2214--2224}, year = {2022}, editor = {Cussens, James and Zhang, Kun}, volume = {180}, series = {Proceedings of Machine Learning Research}, month = {01--05 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v180/yang22c/yang22c.pdf}, url = {https://proceedings.mlr.press/v180/yang22c.html}, abstract = {We attribute the vulnerability of natural language processing models to the fact that similar inputs are converted to dissimilar representations in the embedding space, leading to inconsistent outputs, and we propose a novel robust training method, termed \textit{Fast Triplet Metric Learning (FTML)}. Specifically, we argue that the original sample should have similar representation with its adversarial counterparts and distinguish its representation from other samples for better robustness. To this end, we adopt the triplet metric learning into the standard training to pull words closer to their positive samples (\textit{i.e.}, synonyms) and push away their negative samples (\textit{i.e.}, non-synonyms) in the embedding space. Extensive experiments demonstrate that FTML can significantly promote the model robustness against various advanced adversarial attacks while keeping competitive classification accuracy on original samples. Besides, our method is efficient as it only needs to adjust the embedding and introduces very little overhead on the standard training. Our work shows great potential of improving the textual robustness through robust word embedding.} }
Endnote
%0 Conference Paper %T Robust textual embedding against word-level adversarial attacks %A Yichen Yang %A Xiaosen Wang %A Kun He %B Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2022 %E James Cussens %E Kun Zhang %F pmlr-v180-yang22c %I PMLR %P 2214--2224 %U https://proceedings.mlr.press/v180/yang22c.html %V 180 %X We attribute the vulnerability of natural language processing models to the fact that similar inputs are converted to dissimilar representations in the embedding space, leading to inconsistent outputs, and we propose a novel robust training method, termed \textit{Fast Triplet Metric Learning (FTML)}. Specifically, we argue that the original sample should have similar representation with its adversarial counterparts and distinguish its representation from other samples for better robustness. To this end, we adopt the triplet metric learning into the standard training to pull words closer to their positive samples (\textit{i.e.}, synonyms) and push away their negative samples (\textit{i.e.}, non-synonyms) in the embedding space. Extensive experiments demonstrate that FTML can significantly promote the model robustness against various advanced adversarial attacks while keeping competitive classification accuracy on original samples. Besides, our method is efficient as it only needs to adjust the embedding and introduces very little overhead on the standard training. Our work shows great potential of improving the textual robustness through robust word embedding.
APA
Yang, Y., Wang, X. & He, K.. (2022). Robust textual embedding against word-level adversarial attacks. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 180:2214-2224 Available from https://proceedings.mlr.press/v180/yang22c.html.

Related Material