Elamur Research⌕
☰
home / papers / 2019 / The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction

The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction

Summary

The paper tests how well large pre-trained Transformer models correct grammatical errors (GEC) when they are used only as language models, without any training on annotated data. The authors took the framework of Bryant and Briscoe (2018) and replaced its 5-gram model with BERT, GPT and GPT-2. For each sentence, heuristics build a set of candidate corrections. For prepositions and determiners, these are closed word lists together with a "delete the word" option. For morphology, candidates are taken from the AGID database; for spelling, they are generated with CyHunspell. The model scores the probability of each candidate. A correction is accepted only if it is noticeably better than the original, that is, if it exceeds the threshold τ. Positions in the sentence are traversed with a greedy search. Evaluation was carried out on CoNLL-2014 and FCE with the M2 and ERRANT metrics. GPT and GPT-2 substantially outperformed the n-gram approach and on FCE nearly matched a supervised system. BERT proved the weakest; the authors attribute this to its being trained with a masked rather than a standard language modelling objective. The result matters for languages with few parallel error corpora: candidate sets are relatively easy to build for them, and ready-made language models are already available.

Key points

  • Test data: CoNLL-2014 (1,312 sentences, ~30k tokens, 2 annotators) and FCE (2,715 sentences, ~47k tokens, 1 annotator). The training portions of the corpora were not used, and the dev sets were used only to tune τ ∈ {0, 2, 4, 6, 8}. JFLEG and W&I+LOCNESS were not included because of licensing restrictions.
  • CoNLL-2014, M2 F0.5: GPT-2 — 46.08, GPT — 43.57, BERT — 33.83, the Bryant and Briscoe 5-gram model — 34.09. For comparison, the shared task winner (Felice et al., 2014) scored 37.33, and the best supervised system (Ge et al., 2018) scored 61.34.
  • FCE, M2 F0.5: GPT — 53.52, GPT-2 — 53.48, BERT — 32.53, the n-gram model — 31.22. The supervised system of Yannakoudakis et al. (2017) scored 54.15.
  • Sentence probability with BERT was obtained by masking each word in turn and summing the log probabilities. Candidates not in the vocabulary were replaced with [UNK] so that AGID errors would not distort the scores.
  • Limitations: the method cannot insert missing words (about 20% of all errors) and mainly covers closed word classes. Moreover, because of the predefined candidate set, the system cannot be considered fully unsupervised.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4412 · arXiv 1906.01733

@inproceedings{alikaniotis2019unreasonable,
  title = {{The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction}},
  author = {Alikaniotis, Dimitris and Raheja, Vipul},
  booktitle = {Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2019},
  doi = {10.18653/v1/w19-4412},
  eprint = {1906.01733},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w19-4412}
}

References

34 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Bryant and Ted Briscoe. 2018. Language model based grammatical error correction without annotated training data. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 247–253, New Orleans, Louisiana. Association for Computational… find
  2. Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 15–24, Baltimore, Maryla… find
  3. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Associ… find our card
  4. Michael Gamon, Jianfeng Gao, Chris Brockett, Alexandre Klementiev, William B. Dolan, Dmitriy Belenko, and Lucy Vanderwende. 2008. Using contextual speller techniques and language modeling for ESL error correction. In Proceedings of the Third International Joint Conference on Natural Language Process… find
  5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robin-Kyusong Lee and Gary Geunbae Lee. 2014. POSTECH grammatical error correction system in the CoNLL-2014 shared task. In Proceedings of the Eighteenth Conference on Computational Natural Language Learni… find
  6. Tao Ge, Furu Wei, and Ming Zhou. 2018. Reaching human-level performance in automatic grammatical error correction: An empirical study. CoRR, abs/1807.01270. find
  7. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287. find our card
  8. A. Halevy, P. Norvig, and F. Pereira. 2009. The unreasonable effectiveness of data. IEEE Intelligent Systems, 24(2):8–12. find
  9. Alain Dsilets Matthieu Hermet and Stan Szpakowicz. 2008. Using the web as a linguistic resource to automatically correct lexico-syntactic errors. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Assoc… find
  10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… find
  11. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spai… find
  12. Jianshu Ji, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, and Jianfeng Gao. 2017. A nested attention neural hybrid model for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 753– 76… find
  13. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1– 14, Baltimore, M… find
  14. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2014. The AMU system in the CoNLL-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pag… find
  15. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12, Sofia, Bulgaria. Association for Computa… find
  16. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Phrase-based machine translation is state-of-the-art for automatic grammatical error correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1546–1556, Austin, Texas. Association for Computation… find
  17. Y. Albert Park and Roger Levy. 2011. Automated whole sentence grammar correction using a noisy channel model. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 934–944, Stroudsburg, PA, USA. Association… find
  18. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  20. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  21. Ting-hui Kao, Yu-wei Chang, Hsun-wen Chiu, Tzu-Hsi Yen, Joanne Boisson, Jian-cheng Wu, and Jason S. Chang. 2013. CoNLL-2013 shared task: Grammatical error correction NTHU system description. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 2… find
  22. Alla Rozovskaya and Dan Roth. 2016. Grammatical error correction: Machine translation and classifiers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2205–2215, Berlin, Germany. Association for Computational Linguistics. find
  23. Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017. Grammatical error correction with neural reinforcement learning. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 366–372, Taipei, Taiwan. Asian Federation of Natural… find
  24. Hayashibe, Mamoru Komachi, and Yuji Matsumoto. 2013. NAIST at 2013 CoNLL grammatical error correction shared task. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 26–33, Sofia, Bulgaria. Association for Computational Linguistics. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  26. Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380–386, San Diego, California. Association for Co… find
  27. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a markov random field language model. CoRR, abs/1902.04094. find our card
  28. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  29. Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, and Andrew Y. Ng. 2016. Neural language correction with character-based attention. CoRR, abs/1603.09727. find
  30. Junwen Xing, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Xiaodong Zeng. 2013. UM-checker: A hybrid system for English grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 34–42, Sofia, Bulgaria. Association for… find
  31. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Compu… find
  32. Helen Yannakoudakis, Marek Rei, Øistein E. Andersen, and Zheng Yuan. 2017. Neural sequence-labelling models for grammatical error correction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2795–2806, Copenhagen, Denmark. Association for Computational… find
  33. Xing Yi, Jianfeng Gao, and William B. Dolan. 2008. A web-based English proofing system for English as a second language users. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II. find
  34. Ippei Yoshimoto, Tomoya Kose, Kensuke Mitsuzawa, Keisuke Sakaguchi, Tomoya Mizumoto, Yuta find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/alikaniotis-2019-unreasonable-effectiveness-transformer-language-models/