Elamur Research⌕
☰
home / papers / 2020 / Spelling Error Correction with Soft-Masked BERT

Spelling Error Correction with Soft-Masked BERT

Paper · 2020 · Shaohua Zhang, Haoran Huang, Jicong Liu et al.

Summary

The paper addresses Chinese spelling error correction (CSC) at the character level. In this task, a sentence must be mapped to a corrected sentence of the same length. The prior state of the art fine-tunes BERT to pick a character, or keep the original, at every position. The authors observe that this approach rarely flags errors. They attribute this to masked language model pre-training, where only about 15% of tokens are masked, so the model tends to copy its input.

Their proposed Soft-Masked BERT adds a Bi-GRU detection network. This network estimates an error probability for each character. Each input embedding is then blended with the [MASK] embedding in proportion to that probability before the result enters a BERT correction network. A residual connection links the input embeddings to BERT's final layer, and both networks are trained jointly with combined detection and correction losses.

Experiments use the SIGHAN benchmark and a newly built News Title dataset. Soft-Masked BERT clearly outperforms baselines, including plain BERT fine-tuning. The soft-masking idea could carry over to other detect-then-correct problems.

Key points

  • Detection is handled by a Bi-GRU with a sigmoid output. Soft masking computes e′ = p·e_mask + (1−p)·e, which becomes standard hard masking when p = 1.
  • The correction network is a 12-block BERT with a softmax over all characters. Its training loss linearly combines the detection loss and the correction loss.
  • Training pairs can be produced by injecting errors into clean text using a confusion table.
  • The new News Title dataset consists of news headlines and is roughly ten times larger than earlier CSC datasets.
  • In an analysis of 100 sampled errors, 67% required stronger reasoning and 11% required world knowledge. The remaining 22% had no clear type. An oracle "BERT-Finetune + Force" setup indicates substantial room for further gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China65 works · 19,402 citations🏢ByteDanceBeijing · China18 works · 3,890 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Shaohua Zhang, Haoran Huang, Jicong Liu and Hang Li. 2020. Spelling Error Correction with Soft-Masked BERT. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.82

@inproceedings{zhang2020spelling,
  title = {{Spelling Error Correction with Soft-Masked BERT}},
  author = {Shaohua Zhang, Haoran Huang, Jicong Liu and Hang Li},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.82},
  url = {https://doi.org/10.18653/v1/2020.acl-main.82}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haithem Afli, Zhengwei Qiu, Andy Way, and Páraic Sheridan. 2016. Using smt for ocr error correction of historical texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 962–966. find
  2. Jill Burstein and Martin Chodorow. 1999. Automated essay scoring for nonnative english speakers. In Proceedings of a Symposium on Computer Mediated Language Assessment and Evaluation in Natural Language Processing, pages 68–75. Association for Computational Linguistics. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  4. Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. 2010. A large scale ranker-based system for search query spelling correction. In COL-ING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 358– 36… find
  5. Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651–5655. IEEE. find
  6. Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019. Faspell: A fast, adaptable, simple, powerful chinese spell checker based on dae-decoder paradigm. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 160–169. find
  7. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  9. Bruno Martins and Mário J. Silva. 2004. Spelling correction for search engine queries. In Advances in Natural Language Processing, 4th International Conference, EsTAL 2004, Alicante, Spain, October 20-22, 2004, Proceedings, pages 372–383. find
  10. Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to sighan 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 32–37. find
  11. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018a. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  12. Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018b. A hybrid approach to automatic corpus generation for chinese spelling check. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2517–2527.
  13. Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for chinese spelling check. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5780–5785. find
  14. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
  15. Junjie Yu and Zhenghua Li. 2014. Chinese spelling error detection and correction based on language model, pronunciation, and shape. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 220–223, Wuhan, China. Association for Computational Linguistics. find
  16. Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014. Overview of sighan 2014 bake-off for chinese spelling check. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 126–132. find
  17. Shuiyuan Zhang, Jinhua Xiong, Jianpeng Hou, Qiao Zhang, and Xueqi Cheng. 2015. Hanspeller++: A unified framework for chinese spelling correction. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 38–45. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2020-spelling-error-correction-soft-masked/