Elamur Research⌕
☰
home / papers / 2018 / A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction

A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction

Paper · 2018 · Shamil Chollampatt and Hwee Tou Ng

Summary

Phrase-based SMT systems led grammatical error correction (GEC) for learner English, and earlier neural encoder-decoder models, all built on RNNs, scored well below them. Chollampatt and Ng drop recurrence and use a fully convolutional encoder-decoder, following Gehring et al. (2017). Each side has seven layers with GLU activations, and attention runs in every decoder layer. To deal with rare words, they split them into BPE sub-words. They also initialize the embeddings with fastText vectors pre-trained on large native English corpora, so the model can use character N-gram information. Several models are combined in an ensemble. A rescorer then reranks the candidates using edit-operation counts and an N-gram language model, with weights tuned by MERT. On CoNLL-2014 and JFLEG, the system beats every earlier neural GEC model. It is also the first neural method to beat the leading SMT-based system on both grammaticality and fluency. This closed the gap between neural and statistical GEC, and the authors released their code and models publicly.

Key points

  • Architecture comparison on the CoNLL-2013 test set: the multilayer convolutional model (MLConv) beats a BiLSTM and a single-layer CNN. Seven layers with filter width 3 give each position about 21 tokens of context. The BiLSTM has higher precision but lower recall because its attention is sharper and it tends to copy source words.
  • Embedding initialization (CoNLL-2013, F0.5): random 31.96, word2vec 32.49, fastText 32.97.
  • Earlier neural results on CoNLL-2014 (F0.5) that this work surpasses: Xie et al. 39.97/40.56 (character-level RNN), Schmaltz et al. 41.37 (BiLSTM with edit tags), and Ji et al. 45.15 (hybrid word-character model trained with the non-public CLC corpus).
  • An error-type analysis with ERRANT on CoNLL-2014 shows gains over the SMT state of the art on noun-number, determiner and subject-verb agreement errors. On prepositions the model is roughly level with it. Even the single model without rescoring handles subject-verb agreement better.
  • The code and models are available at github.com/nusnlp/mlconvgec2018.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Grammatical error correction & learner language · 162 works

Cite

Cite as Shamil Chollampatt and Hwee Tou Ng. 2018. A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction. arXiv:1801.08831.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1801.08831 · arXiv 1801.08831

@misc{chollampatt2018multilayer,
  title = {{A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction}},
  author = {Shamil Chollampatt and Hwee Tou Ng},
  year = {2018},
  doi = {10.48550/arxiv.1801.08831},
  eprint = {1801.08831},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1801.08831}
}

References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D.; Cho, K.; and Bengio, Y. 2015. Neural machine translation by jointly learning to align and translate. In Proc.
  2. ICLR. Bengio, Y.; Boulanger-Lewandowski, N.; and Pascanu, R. 2013. Advances in optimizing recurrent networks. In Proc. ICASSP. find
  3. Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017. Enriching word vectors with subword information. TACL 5:135–146.
  4. Bryant, C.; Felice, M.; and Briscoe, T. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proc. ACL.
  5. Chollampatt, S., and Ng, H. T. 2017. Connecting the dots: Towards human-level grammatical error correction. In Proc. of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. find
  6. Chollampatt, S.; Taghipour, K.; and Ng, H. T. 2016. Neural network translation models for grammatical error correction. In Proc. IJCAI.
  7. Dahlmeier, D., and Ng, H. T. 2012. Better evaluation for grammatical error correction. In Proc. NAACL. find
  8. Dahlmeier, D.; Ng, H. T.; and Ng, E. J. F. 2012. NUS at the HOO 2012 shared task. In Proc. of the 7th Workshop on the Innovative Use of NLP for Building Educational Applications. find
  9. Dahlmeier, D.; Ng, H. T.; and Wu, S. M. 2013. Building a large annotated corpus of learner English: The NUS Corpus of Learner English. In Proc. of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.
  10. Dauphin, Y. N.; Fan, A.; Auli, M.; and Grangier, D. 2016. Language modeling with gated convolutional networks. arXiv preprint arXiv:1612.08083.
  11. Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin, Y. N. 2017. Convolutional sequence to sequence learning. In Proc.
  12. ICML. Ji, J.; Wang, Q.; Toutanova, K.; Gong, Y.; Truong, S.; and Gao, J. 2017. A nested attention neural hybrid model for grammatical error correction. In Proc. ACL. find
  13. Junczys-Dowmunt, M., and Grundkiewicz, R. 2016. Phrase-based machine translation is state-of-the-art for automatic grammatical error correction. In Proc. EMNLP. find
  14. Luong, M.-T., and Manning, C. D. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proc.
  15. ACL. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Proc.
  16. NIPS. Mizumoto, T.; Komachi, M.; Nagata, M.; and Matsumoto, Y. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proc. find
  17. IJCNLP. Napoles, C.; Sakaguchi, K.; Post, M.; and Tetreault, J. 2015. Ground truth for grammatical error correction metrics. In Proc. ACL/IJCNLP. find
  18. Napoles, C.; Sakaguchi, K.; and Tetreault, J. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proc. find
  19. EACL. Ng, H. T.; Wu, S. M.; Briscoe, T.; Hadiwinoto, C.; Susanto, R. H.; and Bryant, C. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proc. CoNLL: Shared Task. find
  20. Och, F. J. 2003. Minimum error rate training in statistical machine translation. In Proc. find
  21. ACL. Rozovskaya, A.; Chang, K.-W.; Sammons, M.; Roth, D.; and Habash, N. 2014. The Illinois-Columbia system in the CoNLL-2014 shared task. In Proc. CoNLL: Shared Task.
  22. Schmaltz, A.; Kim, Y.; Rush, A. M.; and Shieber, S. 2017. Adapting sequence models for sentence correction. In Proc. EMNLP.
  23. Sennrich, R.; Haddow, B.; and Birch, A. 2016. Neural machine translation of rare words with subword units. In Proc.
  24. ACL. Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR 15(1):1929– 1958.
  25. Xie, Z.; Avati, A.; Arivazhagan, N.; Jurafsky, D.; and Ng, A. Y. 2016. Neural language correction with character-based attention. arXiv preprint arXiv:1603.09727.
  26. Yuan, Z., and Briscoe, T. 2016. Grammatical error correction using neural machine translation. In Proc. NAACL.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chollampatt-2018-multilayer-convolutional-encoder-decoder-neural/