Evaluating the Robustness of Biomedical Concept Normalization

Sinchani Chakraborty, Harsh Raj, Srishti Gureja, Tanmay Jain, Atif Hassan, Sayantan Basu
Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop, PMLR 203:63-73, 2023.

Abstract

Biomedical concept normalization involves linking entity mentions in text to standard concepts in knowledge bases. It aids in resolving challenges to standardising ambiguous, variable terms in text or handling missing links. Therefore, it is one of the essential tasks of text mining that helps in effective information access and finds its utility in biomedical decision-making. Pre-trained language models (e.g., BERT) achieve impressive performance on this task. It has been observed that such models are insensitive to word order permutations and vulnerable to adversarial attacks on tasks like Text Classification, Natural Language Inference. However, the effect of such attacks is unknown for the task of Normalization, especially in the biomedical domain. In this paper, we propose heuristics-based Input Transformations (word level modifications and word order variations) and Adversarial Attacks to study the robustness of BERT-based normalization models across various datasets consisting of different biomedical entity types. We conduct experiments across three datasets: NCBI disease, BC5CDR Disease, and BC5CDR Chemical. We observe that for input transformations, pre-trained models often fail to detect invalid input. On the other hand, our proposed adversarial attacks that add imperceptible perturbations, result in affecting the ranking of a concept list for a given mention (or vice versa). We also generate natural adversarial examples that lead to performance degradation of  30% in the F1-score. Additionally, we explore existing mitigation strategies to help a model recognize invalid inputs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v203-chakraborty23a, title = {Evaluating the Robustness of Biomedical Concept Normalization}, author = {Chakraborty, Sinchani and Raj, Harsh and Gureja, Srishti and Jain, Tanmay and Hassan, Atif and Basu, Sayantan}, booktitle = {Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop}, pages = {63--73}, year = {2023}, editor = {Albalak, Alon and Zhou, Chunting and Raffel, Colin and Ramachandran, Deepak and Ruder, Sebastian and Ma, Xuezhe}, volume = {203}, series = {Proceedings of Machine Learning Research}, month = {03 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v203/chakraborty23a/chakraborty23a.pdf}, url = {https://proceedings.mlr.press/v203/chakraborty23a.html}, abstract = {Biomedical concept normalization involves linking entity mentions in text to standard concepts in knowledge bases. It aids in resolving challenges to standardising ambiguous, variable terms in text or handling missing links. Therefore, it is one of the essential tasks of text mining that helps in effective information access and finds its utility in biomedical decision-making. Pre-trained language models (e.g., BERT) achieve impressive performance on this task. It has been observed that such models are insensitive to word order permutations and vulnerable to adversarial attacks on tasks like Text Classification, Natural Language Inference. However, the effect of such attacks is unknown for the task of Normalization, especially in the biomedical domain. In this paper, we propose heuristics-based Input Transformations (word level modifications and word order variations) and Adversarial Attacks to study the robustness of BERT-based normalization models across various datasets consisting of different biomedical entity types. We conduct experiments across three datasets: NCBI disease, BC5CDR Disease, and BC5CDR Chemical. We observe that for input transformations, pre-trained models often fail to detect invalid input. On the other hand, our proposed adversarial attacks that add imperceptible perturbations, result in affecting the ranking of a concept list for a given mention (or vice versa). We also generate natural adversarial examples that lead to performance degradation of  30% in the F1-score. Additionally, we explore existing mitigation strategies to help a model recognize invalid inputs.} }
Endnote
%0 Conference Paper %T Evaluating the Robustness of Biomedical Concept Normalization %A Sinchani Chakraborty %A Harsh Raj %A Srishti Gureja %A Tanmay Jain %A Atif Hassan %A Sayantan Basu %B Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop %C Proceedings of Machine Learning Research %D 2023 %E Alon Albalak %E Chunting Zhou %E Colin Raffel %E Deepak Ramachandran %E Sebastian Ruder %E Xuezhe Ma %F pmlr-v203-chakraborty23a %I PMLR %P 63--73 %U https://proceedings.mlr.press/v203/chakraborty23a.html %V 203 %X Biomedical concept normalization involves linking entity mentions in text to standard concepts in knowledge bases. It aids in resolving challenges to standardising ambiguous, variable terms in text or handling missing links. Therefore, it is one of the essential tasks of text mining that helps in effective information access and finds its utility in biomedical decision-making. Pre-trained language models (e.g., BERT) achieve impressive performance on this task. It has been observed that such models are insensitive to word order permutations and vulnerable to adversarial attacks on tasks like Text Classification, Natural Language Inference. However, the effect of such attacks is unknown for the task of Normalization, especially in the biomedical domain. In this paper, we propose heuristics-based Input Transformations (word level modifications and word order variations) and Adversarial Attacks to study the robustness of BERT-based normalization models across various datasets consisting of different biomedical entity types. We conduct experiments across three datasets: NCBI disease, BC5CDR Disease, and BC5CDR Chemical. We observe that for input transformations, pre-trained models often fail to detect invalid input. On the other hand, our proposed adversarial attacks that add imperceptible perturbations, result in affecting the ranking of a concept list for a given mention (or vice versa). We also generate natural adversarial examples that lead to performance degradation of  30% in the F1-score. Additionally, we explore existing mitigation strategies to help a model recognize invalid inputs.
APA
Chakraborty, S., Raj, H., Gureja, S., Jain, T., Hassan, A. & Basu, S.. (2023). Evaluating the Robustness of Biomedical Concept Normalization. Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop, in Proceedings of Machine Learning Research 203:63-73 Available from https://proceedings.mlr.press/v203/chakraborty23a.html.

Related Material