Elamur Research⌕
☰
home / papers / 2016 / Grammatical error correction using neural machine translation

Grammatical error correction using neural machine translation

Paper · 2016 · Zheng Yuan and Ted Briscoe

Summary

The authors were the first to apply neural machine translation (NMT) to grammatical error correction (GEC) in English texts written by non-native speakers. The main obstacle is NMT's limited vocabulary. Rare words, and especially misspelled words, become the UNK token, so the system can neither correct them nor leave them unchanged. To address this, a two-stage scheme is proposed. First, unsupervised aligners (GIZA++ with hints from METEOR) identify the word in the input sentence from which each UNK in the output originates. Then this word is translated by a word-level model built on IBM Model 4. Unlike the method of Luong et al., the training data do not need to be re-annotated, and a single NMT model is sufficient. The RNNsearch model was trained on approximately two million parallel sentence pairs from the CLC. On the FCE test set, the best system outperformed a phrase-based SMT system on all three metrics. On CoNLL-2014, without any training on NUCLE, it achieved a new state-of-the-art result. Thus, neural GEC transfers well to other data and is competitive with SMT.

Key points

  • Data: FCE (part of the CLC, 1,244 exam scripts) together with additional examples from the CLC. In total, this amounts to 1,965,727 sentence pairs. Because of errors in the input texts, their vocabulary is considerably larger than that of the corrected texts: 248,028 versus 143,852 unique words.
  • The RNNsearch model has a bidirectional RNN encoder and an attention-based decoder. The target vocabulary is limited to 30K; for the source vocabulary, 30K, 50K and 80K were tested. Training used sentences of up to 100 words, and each model was trained for about 5 days on a Tesla K20.
  • FCE test: the best NMT configuration, 30K-30K + GIZA++ + METEOR, achieved an F0.5 of 53.49%, an I-measure of 3.94% and a GLEU of 71.16%. The SMT baseline scores 52.90 / 2.87 / 70.15. Without UNK replacement, NMT falls behind SMT: F0.5 of 46.10–49.17 and a negative I-measure.
  • UNK replacement helps models with a small source vocabulary the most: the larger the vocabulary, the smaller the gain. Information from METEOR further improves the GIZA++ alignment.
  • CoNLL-2014 (without alternative answers): F0.5 of 39.90%. This is higher than the combined system of Susanto et al. (39.39%) and the top three systems of the shared task: CAMB 37.33, CUUI 36.79, AMU 35.01. Among the compared systems, it also has the highest GLEU (65.59) and I-measure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1042

@inproceedings{yuan2016grammatical,
  title = {{Grammatical error correction using neural machine translation}},
  author = {Zheng Yuan and Ted Briscoe},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1042},
  url = {https://doi.org/10.18653/v1/n16-1042}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR, abs/1409.0473. ↗
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72.
  3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 77–80.
  4. Chris Brockett, William B. Dolan, and Michael Gamon. 2006. Correcting ESL errors using phrasal SMT techniques. In Proceedings of the COLING/ACL 2006, pages 249–256.
  5. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 19(2):263–311. find
  6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724–1734. ↗
  7. Daniel Dahlmeier and Hwee Tou Ng. 2011. Grammatical error correction with alternating structure optimization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 915–923. find
  8. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572. find
  9. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner english: the NUS Corpus of Learner English. In Proceedings of the 8th Workshop on Innovative Use of NLP for Building Educational Applications, pages 22– 31.
  10. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: an open source toolkit for handling large scale language models. In Proceedings of the 9th Annual Conference of the International Speech Communication Association, pages 1618–1621. find
  11. Mariano Felice and Ted Briscoe. 2015. Towards a standard evaluation method for grammatical error detection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 578–587. find
  12. Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, pages 15–24. find
  13. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1–10. ↗
  14. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2014. The AMU system in the CoNLL-2014 shared task: grammatical error correction by data-intensive and feature-rich statistical machine translation. In Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, pages 25– 33. find
  15. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709.
  16. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 181– 184. find
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 177– 180. find
  18. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press. ↗
  19. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the Rare Word Problem in Neural Machine Translation. In Proceedings of the ACL-IJCNLP, pages 11–19. ↗
  20. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 588–593. find
  21. Graham Neubig, Taro Watanabe, Eiichiro Sumita, Shinsuke Mori, and Tatsuya Kawahara. 2011. An unsupervised model for joint phrase alignment and extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 632–641. find
  22. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12. find
  23. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, pages 1–14. find
  24. Diane Nicholls. 2003. The Cambridge Learner Corpus - error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 Conference, pages 572–581. find
  25. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. ↗
  27. Alla Rozovskaya and Dan Roth. 2011. Algorithm selection and model adaptation for ESL correction tasks. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 924–933. find
  28. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. 2014. The Illinois-Columbia System in the CoNLL-2014 Shared Task. In Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, pages 34–42.
  29. Hendy Raymond Susanto, Peter Phandi, and Tou Hwee Ng. 2014. System combination for grammatical error correction. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 951–962. find
  30. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104–3112. ↗
  31. Joel R. Tetreault and Martin Chodorow. 2008. The ups and downs of preposition error detection in ESL writing. In Proceedings of the 22nd International Conference on Computational Linguistics, volume 1, pages 865–872.
  32. Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In Proceedings of the 16th International Conference on Computational linguistics, volume 2, pages 836–841. find
  33. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189.
  34. Zheng Yuan and Mariano Felice. 2013. Constrained grammatical error correction using statistical machine translation. In Proceedings of the 17th Conference on Computational Natural Language Learning: Shared Task, pages 52–61. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2016-grammatical-error-correction-neural-machine/