Elamur Research⌕
☰
home / papers / 2016 / Neural Language Correction with Character-Based Attention

Neural Language Correction with Character-Based Attention

Paper · 2016 · Ziang Xie, Anand Avati, Naveen Arivazhagan et al.

Summary

The paper addresses automatic correction of English learners' writing. Earlier systems either relied on per-error-type classifiers, which are precise but miss redundancy and unidiomatic wording, or on phrase-based machine translation, which handles misspellings poorly. The authors cast correction as character-level sequence transduction. Their encoder-decoder RNN uses a pyramidal bidirectional GRU encoder and an attention-based decoder, so it needs no fixed word vocabulary and avoids OOV problems. Decoding runs a beam search that mixes the network's scores with a large n-gram language model. A separate edit classifier then removes doubtful changes to raise precision. Training also used extra sentences with artificially injected errors. The model was trained mainly on the noisy Lang-8 Corpus and evaluated on the CoNLL 2014 Shared Task, where it reached a state-of-the-art F0.5. It was especially strong on orthographic, redundancy and collocation errors. The work shows that one neural model can fix many error types in noisy user text without task-specific features. Its main limitation is that it uses only single-sentence context.

Key points

  • Architecture: a 3-layer pyramidal bi-GRU encoder, which shortens the sequence 4×, and a 3-layer GRU decoder. Both use 400 hidden units and a 98-symbol character vocabulary (printable ASCII plus start, end and unknown tokens).
  • Language model: a Kneser-Ney 5-gram model built with KenLM on a slice of Common Crawl, about 2.2 billion n-grams after pruning. Its score enters beam search at word boundaries.
  • Precision control: an MLP edit classifier using edit-distance and GloVe features keeps only edits above a probability threshold. On CoNLL 2014 the system's precision was 49.24, compared with 41.78 for the best of the top 10 challenge teams.
  • On the Lang-8 test set, test BLEU rose from 59.54 with no edits to 61.63 for the RNN and 61.70 with the LM added.
  • Recall on the Mec error type (spelling, punctuation, capitalization) was 37.17, beaten only by the rule-based RAC team. The system also ranked 2nd on Rloc- (redundancy) and 3rd on Wci (collocations).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky and Andrew Y. Ng. 2016. Neural Language Correction with Character-Based Attention. arXiv:1603.09727.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.09727 · arXiv 1603.09727

@misc{xie2016neural,
  title = {{Neural Language Correction with Character-Based Attention}},
  author = {Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky and Andrew Y. Ng},
  year = {2016},
  doi = {10.48550/arxiv.1603.09727},
  eprint = {1603.09727},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.09727}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  2. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). find
  3. William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211. find
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
  5. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In North American Chapter of the Association for Computational Linguistics (NAACL). find
  6. Mariano Felice and Zheng Yuan. 2014. Generating artificial errors for grammatical error correction. In EACL. find
  7. Mariano Felice, Zheng Yuan, istein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. find
  8. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  9. Na-Rae Han, Martin Chodorow, and Claudia Leakcock. 2006. Detecting errors in english article usage by non-native speakers. Natural Language Engineering. find
  10. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567.
  11. Kenneth Heafield and Alon Lavie. 2010. Cmu multi-engine machine translation for wmt 2010. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT ’10. find
  12. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In ACL-HLT, pages 690–696, Sofia, Bulgaria.
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  14. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2014. The amu system in the conll-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. find
  15. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Empirical Methods in Natural Language Processing (EMNLP).
  16. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  17. Kyusong Lee and Gary Geunbae Lee. 2014. Postech grammatical error correction system in the conll-2014 shared task. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. find
  18. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. Character-based neural machine translation. arXiv preprint arXiv:1511.04586.
  19. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206.
  20. Andrew L. Maas, Ziang Xie, Dan Jurafsky, and Andrew Y. Ng. 2015. Lexicon-free conversational speech recognition with neural networks. In Proceedings the North American Chapter of the Association for Computational Linguistics (NAACL). find
  21. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations.
  22. Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning sns for automated japanese error correction of second language learners. In International Joint Conference on Natural Language Processing (IJCNLP). find
  23. Tomoya Mizumoto, Yuta Hayashibe, Mamoru Komachi, Masaaki Nagata, and Yuji Matsuomto. 2012. The effect of learner corpus size in grammatical error correction of ESL writings. In International Conference on Computational Linguistics. find
  24. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. find
  25. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. find
  26. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In International Conference on Machine learning (ICML). find
  27. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP).
  28. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on. find
  29. Alla Rozovskaya and Dan Roth. 2010. Generating confusion sets for context-sensitive error correction. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language (EMNLP). find
  30. Alla Rozovskaya, Mark Sammons, and Dan Roth. 2012. The ui system in the hoo 2012 shared task on error correction. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP. find
  31. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. 2014. The illinois-columbia system in the conll-2014 shared task. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task.
  32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research.
  33. Raymond Hendy Susanto. 2014. Systems combination for grammatical error correction. In Empirical Methods in Natural Language Processing (EMNLP). find
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Neural Information Processing Systems (NIPS).
  35. Toshikazu Tajiri, Mamoru Komachi, and Yuji Matsumoto. 2012. Tense and aspect error correction for esl learners using global context. In Association for Computational Linguistics: Short Papers.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2016-neural-language-correction-character-based/