CoSPA: An improved masked language model with copy mechanism for Chinese spelling correction

Shoujian Yang, Lian Yu
Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, PMLR 180:2225-2234, 2022.

Abstract

Existing BERT-based models for Chinese spelling correction (CSC) have three issues. 1) Bert tends to rectify a correct low-frequency collocation into a high-frequency one and leads to over-correcting. 2) It fails to completely detect phonic or morphological errors by the current learned similarity knowledge between Chinese characters, and the recall rate still has room to improve. 3) Two-dimensional glyph information of Chinese characters is overlooked and some morphological misused characters may be difficult to detect. This paper proposes a hybrid approach, CoSPA, to address these issues. 1) This paper proposes an alterable copy mechanism to alleviate over-correcting by jointly learning to copy a correct character from input sentence, or generate a character from BERT. No method has used copy mechanism in BERT for CSC. 2) The attention mechanism is further applied on the phonic and shape representation of each character at the output layer. 3) Shape representation is enhanced by mining character glyph with ResNet, and fused with stroke representation via an adaptive gating unit. The experimental results show that CoSPA outperforms the previous state-of-the-art methods on SIGHAN2015 datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v180-yang22d, title = {CoSPA: An improved masked language model with copy mechanism for Chinese spelling correction}, author = {Yang, Shoujian and Yu, Lian}, booktitle = {Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence}, pages = {2225--2234}, year = {2022}, editor = {Cussens, James and Zhang, Kun}, volume = {180}, series = {Proceedings of Machine Learning Research}, month = {01--05 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v180/yang22d/yang22d.pdf}, url = {https://proceedings.mlr.press/v180/yang22d.html}, abstract = {Existing BERT-based models for Chinese spelling correction (CSC) have three issues. 1) Bert tends to rectify a correct low-frequency collocation into a high-frequency one and leads to over-correcting. 2) It fails to completely detect phonic or morphological errors by the current learned similarity knowledge between Chinese characters, and the recall rate still has room to improve. 3) Two-dimensional glyph information of Chinese characters is overlooked and some morphological misused characters may be difficult to detect. This paper proposes a hybrid approach, CoSPA, to address these issues. 1) This paper proposes an alterable copy mechanism to alleviate over-correcting by jointly learning to copy a correct character from input sentence, or generate a character from BERT. No method has used copy mechanism in BERT for CSC. 2) The attention mechanism is further applied on the phonic and shape representation of each character at the output layer. 3) Shape representation is enhanced by mining character glyph with ResNet, and fused with stroke representation via an adaptive gating unit. The experimental results show that CoSPA outperforms the previous state-of-the-art methods on SIGHAN2015 datasets.} }
Endnote
%0 Conference Paper %T CoSPA: An improved masked language model with copy mechanism for Chinese spelling correction %A Shoujian Yang %A Lian Yu %B Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2022 %E James Cussens %E Kun Zhang %F pmlr-v180-yang22d %I PMLR %P 2225--2234 %U https://proceedings.mlr.press/v180/yang22d.html %V 180 %X Existing BERT-based models for Chinese spelling correction (CSC) have three issues. 1) Bert tends to rectify a correct low-frequency collocation into a high-frequency one and leads to over-correcting. 2) It fails to completely detect phonic or morphological errors by the current learned similarity knowledge between Chinese characters, and the recall rate still has room to improve. 3) Two-dimensional glyph information of Chinese characters is overlooked and some morphological misused characters may be difficult to detect. This paper proposes a hybrid approach, CoSPA, to address these issues. 1) This paper proposes an alterable copy mechanism to alleviate over-correcting by jointly learning to copy a correct character from input sentence, or generate a character from BERT. No method has used copy mechanism in BERT for CSC. 2) The attention mechanism is further applied on the phonic and shape representation of each character at the output layer. 3) Shape representation is enhanced by mining character glyph with ResNet, and fused with stroke representation via an adaptive gating unit. The experimental results show that CoSPA outperforms the previous state-of-the-art methods on SIGHAN2015 datasets.
APA
Yang, S. & Yu, L.. (2022). CoSPA: An improved masked language model with copy mechanism for Chinese spelling correction. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 180:2225-2234 Available from https://proceedings.mlr.press/v180/yang22d.html.

Related Material