Elamur Research⌕
☰
home / papers / 2016 / Compositional Sequence Labeling Models for Error Detection in Learner Writing

Compositional Sequence Labeling Models for Error Detection in Learner Writing

Paper · 2016 · Marek Rei and Helen Yannakoudakis

Summary

The paper treats grammatical error detection in learner English as a task of its own rather than a side effect of error correction. Each token is labeled as correct or incorrect given its sentence context. The authors compare six neural composition architectures for building token-in-context representations: convolutional, bidirectional Elman-type recurrent, and bidirectional LSTM networks, each in single- and multi-layer form. A CRF++ baseline is included for reference. Experiments use the FCE-public corpus, with error spans converted into token-level labels.

The best design is a multi-layer bidirectional LSTM, which conditions every label on the whole sentence. With extra training data, it surpasses all CoNLL-14 shared task participants on error detection. This is notable because it is a single model, whereas competing systems are mostly ensembles of several components.

When its token probabilities are added as a feature to an existing automated essay scoring system, the scorer improves. With the larger training set, its correlation with gold scores approaches or exceeds that of human annotators. This suggests error detection is useful for real learner-feedback tools.

Key points

  • Data split: FCE-public has 77 error types. The authors use 30,953 training sentences (2,222 of them held out for development) and 2,720 test sentences. For a missing word, the error label goes on the token that follows the gap.

  • CoNLL-14 result: trained on all available data, the Bi-LSTM reaches the highest F0.5 on both CoNLL-14 test annotations. This is roughly a 3% absolute gain over the previous best. The best correction system (P1+P2+S1+S2) was not the best detector (CAMB).

  • Training data: most of the gain from additional data came from about 8 million tokens written by learners of mixed proficiency. Adding more text from higher-proficiency writers helped only slightly.

  • Essay scoring: the Bi-LSTM output was added to the deployed SAT scorer (Andersen et al., 2013).

    SystemPearson rSpearman ρ
    SAT alone75.176.0
    SAT + Bi-LSTM (FCE-public only)76.077.0
    SAT + Bi-LSTM (full data)78.079.9
    Human annotators79.679.2

  • Error reduction: the full-data combination cuts the gap to human performance on Pearson's correlation by 64% (relative error reduction) and beats humans on Spearman's correlation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Marek Rei and Helen Yannakoudakis. 2016. Compositional Sequence Labeling Models for Error Detection in Learner Writing. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1112 · arXiv 1607.06153

@inproceedings{rei2016compositional,
  title = {{Compositional Sequence Labeling Models for Error Detection in Learner Writing}},
  author = {Marek Rei and Helen Yannakoudakis},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1112},
  eprint = {1607.06153},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1112}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Øistein E. Andersen, Helen Yannakoudakis, Fiona Barker, and Tim Parish. 2013. Developing and testing a self-assessment and tutoring system. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications. 53rd Annual Meeting of the Association for Computational Ling… find
  2. Ciprian Chelba, Tomáš Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. In arXiv preprint.
  3. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation.
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 20… find
  5. Martin Chodorow and Claudia Leacock. 2000. An unsupervised method for detecting grammatical errors. In Proceedings of the first conference on North American chapter of the Association for Computational Linguistics. find
  6. Martin Chodorow, Joel R. Tetreault, and Na-Rae Han. 2007. Detection of grammatical errors involving prepositions. In Proceedings of the 4th ACL-SIGSEM Workshop on Prepositions. find
  7. Martin Chodorow, Markus Dickinson, Ross Israel, and Joel Tetreault. 2012. Problems in evaluating grammatical error detection systems. In COLING 2012. find
  8. George. E. Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20. find
  9. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The NUS corpus of learner English. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications. find
  10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations.
  11. Robert Dale and Adam Kilgarriff. 2011. find
  12. Helping Our Own: The HOO 2011 Pilot Shared Task. In Proceedings of the 13th European Workshop on Natural Language Generation.
  13. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model Yoshua. Journal of Machine Learning Research, 3.
  14. Robert Dale, Ilya Anisimoff, and George Narroway. 2012. find
  15. HOO 2012: A report on the Preposition and Determiner Error Correction Shared Task. In The Seventh Workshop on Building Educational Applications Using NLP.
  16. Christopher Bryant and Hwee Tou Ng. 2015. How Far are We from Fully Automatic High Quality Grammatical Error Correction? In Proceedings of the Jeffrey L. Elman. 1990. Finding structure in time. Cognitive science, 14(2). find
  17. Mariano Felice and Ted Briscoe. 2015. Towards a standard evaluation method for grammatical error detection and correction. In The 2015 Annual Conference of the North American Chapter of the ACL. find
  18. Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. Conference on Computational Natural Language Learning: Shared Task (CoNLL-2014). find
  19. Michael Gamon. 2011. High-Order Sequence Modeling for Language Learner Error Detection. Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications. find
  20. Alex Graves, Navdeep Jaitly, and Abdel Rahman Mohamed. 2013. Hybrid speech recognition with Deep Bidirectional LSTM. In IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU 2013). find
  21. Na-Rae Han, Martin Chodorow, and Claudia Leacock. 2004. Detecting Errors in English Article Usage with a Maximum Entropy Classifier Trained on a Large, Diverse Corpus. Proceedings of the 4th International Conference on Language Resources and Evaluation. find
  22. Na-Rae Han, Martin Chodorow, and Claudia Leacock. 2006. Detecting errors in English article usage by non-native speakers. Natural Language Engineering, 12. find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-term Memory. Neural Computation, 9. find
  24. Ozan Irsoy and Claire Cardie. 2014. Opinion Mining with Deep Recurrent Neural Networks. In EMNLP-2014. find
  25. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2014. The AMU System in the CoNLL-2014 Shared Task: Grammatical Error Correction by Data-Intensive and Feature-Rich Statistical Machine Translation. Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL-… find
  26. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013).
  27. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: a Method for Stochastic Optimization. In International Conference on Learning Representations. find
  28. Ekaterina Kochmar and Ted Briscoe. 2014. Detecting Learner Errors in the Choice of Content Words Using Compositional Distributional Semantics. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. find
  29. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning. Citeseer. find
  30. Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel R. Tetreault. 2014. Automated Grammatical Error Detection for Language Learners: Second Edition. find
  31. John Lee and Stephanie Seneff. 2008. Correcting misuse of verb forms. In Proceedings of the 46th Annual Meeting of the ACL. find
  32. Mnih-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  33. Tomáš Mikolov, Stefan Kombrink, Anoop Deoras, Lukáš Burget, and Jan Černocký. 2011. RNNLM-Recurrent neural network language modeling toolkit. In ASRU 2011 Demo Session. find
  34. Tomáš Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. Proceedings of the International Conference on Learning Representations (ICLR 2013).
  35. Ryo Nagata and Kazuhide Nakatani. 2010. Evaluating performance of grammatical error detection to maximize learning effect. find
  36. Coling 2010: Poster Volume. find
  37. Hwee Tou Ng, Yuanbin Wu, and Christian Hadiwinoto. 2013. The CoNLL-2013 Shared Task on Grammatical Error Correction. Computational Natural Language Learning (CoNLL), Shared Task. find
  38. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. find
  39. Diane Nicholls. 2003. The Cambridge Learner Corpus - error coding and analysis for lexicography and ELT. Proceedings of the Corpus Linguistics 2003 Conference. find
  40. Alla Rozovskaya and Dan Roth. 2010. Training Paradigms for Correcting Errors in Grammar and Usage. find
  41. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  42. Alla Rozovskaya, Mark Sammons, Joshua Gioja, and Dan Roth. 2011. University of Illinois System in HOO Text Correction Shared Task. Proceedings of the 13th European Workshop on Natural Language Generation (ENLG). find
  43. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, and Dan Roth. 2013. The University of Illinois System in the CoNLL-2013 Shared Task. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task. find
  44. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. 2014. The Illinois-Columbia System in the CoNLL-2014 Shared Task. Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL-2014). find
  45. Raymond Hendy Susanto, Peter Phandi, and Hwee Tou Ng. 2014. System Combination for Grammatical Error Correction. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP-2014). find
  46. Joel R. Tetreault and Martin Chodorow. 2008. The Ups and Downs of Preposition Error Detection in ESL Writing. Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008).
  47. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems.
  48. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rei-2016-compositional-sequence-labeling-models-error/