Elamur Research⌕
☰
home / papers / 2017 / Artificial Error Generation with Machine Translation and Syntactic Patterns

Artificial Error Generation with Machine Translation and Syntactic Patterns

Summary

Automated detection of writing errors is limited by the small amount of manually annotated learner data. Rei, Felice, Yuan and Briscoe (Cambridge, ALTA Institute) address this by synthesising erroneous text from correct sentences. They compare two supervised approaches, both trained on the public FCE training set, that cover all error types rather than a handful.

  • MT approach: a phrase-based SMT system (Pialign, Moses, IRSTLM 5-gram LM) learns to "translate" corrected sentences back into learner-like text.
  • Pattern approach: short word/POS transformation patterns are mined from aligned corrections and inserted into clean text. This extends Felice and Yuan (2014).

Generated sentences are aligned to their sources with Levenshtein distance to obtain token-level labels. These labels are added to the real annotations for training a bidirectional LSTM sequence labeler. Every artificial-data setup significantly beat training on annotation alone. Combining both generators performed best on FCE and both CoNLL 2014 annotations, improving F0.5 by about 4.3 points absolute on FCE. The approach needs no extra annotated data and could also help anonymise private corpora.

Key points

  • Error coverage: prior AEG work targeted narrow error categories. The five types used by FY14 account for only 35.74% of errors in the CoNLL 2014 training data.

  • Pattern resource: keeping patterns seen at least 5 times gave 35,625 patterns. POS tags came from RASP with the CLAWS2 tagset.

  • Source text for error insertion:

    • the corrected FCE training set (450K tokens);
    • English Vocabulary Profile example sentences (270K tokens).

    In-domain text worked better than larger general corpora.

  • F0.5 results (annotation only → Ann+PAT+MT):

    Test setAnnotation onlyAnn+PAT+MT
    FCE test44.8449.11
    CoNLL-14 TEST116.5221.87
    CoNLL-14 TEST224.5730.13

    Significance was assessed with the Approximate Randomisation Test.

  • Multiple versions: training on several generated versions of the same text (e.g., SMT n-best lists) further raised development-set F0.5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

34 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 32 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.05236 · arXiv 1707.05236

@misc{rei2017artificial,
  title = {{Artificial Error Generation with Machine Translation and Syntactic Patterns}},
  author = {Rei, Marek and Felice, Mariano and Yuan, Zheng and Briscoe, Ted},
  year = {2017},
  doi = {10.48550/arxiv.1707.05236},
  eprint = {1707.05236},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.05236}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The Second Release of the RASP System. In Proceedings of the COLING/ACL on Interactive presentation sessions.. Association for Computational Linguistics, Sydney, Australia, July, pages 77–80. https://doi.org/10.3115/1225403.1225423. ↗
  2. Chris Brockett, William B. Dolan, and Michael Gamon. 2006. Correcting ESL errors using phrasal SMT techniques. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics https://doi.org/10.3115/1220175.1220207. ↗
  3. Aoife Cahill, Nitin Madnani, Joel Tetreault, and Diane Napolitano. 2013. Robust Systems for Preposition Error Correction Using Wikipedia Revisions. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. http://www.aclweb.org/anthology/N13-1055. ↗
  4. Paul Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, MA. find
  5. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: an open source toolkit for handling large scale language models. In Proceedings of the 9th Annual Conference of the International Speech Communication Association. find
  6. Mariano Felice. 2016. Artificial error generation for translation-based grammatical error correction. Technical Report UCAM-CL-TR-895, University of Cambridge, Computer Laboratory. http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-895.pdf. ↗
  7. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in esl sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. ↗
  8. The COLING 2016 Organizing Committee, Osaka, Japan, pages 825–835. http://aclweb.org/anthology/C16-1079. ↗
  9. Mariano Felice and Zheng Yuan. 2014. Generating artificial errors for grammatical error correction. Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014) . find
  10. Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task. find
  11. Jennifer Foster and Øistein E. Andersen. 2009. GenERRate: generating errors for use in grammatical error detection. In Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications. http://dl.acm.org/citation.cfm?id=1609855. ↗
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-term Memory. Neural Computation 9. https://doi.org/10.1.1.56.7752. ↗
  13. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. find
  14. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions. find
  15. Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Exploring grammatical error correction with not-so-crummy machine translation. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP. Association for Computational Linguistics, Montréal, Canada, pages 44–53. http://www.aclweb.org/anthology/W12-2005. ↗
  16. Graham Neubig, Taro Watanabe, Eiichiro Sumita, Shinsuke Mori, and Tatsuya Kawahara. 2011. An unsupervised model for joint phrase alignment and extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find
  17. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. http://www.aclweb.org/anthology/W/W14/W14-1701. ↗
  18. Eric W. Noreen. 1989. Computer Intensive Methods for Testing Hypotheses. Wiley, New York. find
  19. Marek Rei. 2017. Semi-supervised Multitask Learning for Sequence Labeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL-2017). find
  20. Marek Rei and Helen Yannakoudakis. 2016. Compositional Sequence Labeling Models for Error Detection in Learner Writing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. https://aclweb.org/anthology/P/P16/P16-1112.pdf. ↗
  21. Alla Rozovskaya and Dan Roth. 2010. Generating confusion sets for context-sensitive error correction. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing http://dl.acm.org/citation.cfm?id=1870752. ↗
  22. Alla Rozovskaya and Dan Roth. 2011. Algorithm Selection and Model Adaptation for ESL Correction Tasks. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011) http://www.aclweb.org/anthology-new/P/P11/P11-1093.pdf. ↗
  23. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. http://www.aclweb.org/anthology/P11-1019. ↗
  24. Zheng Yuan and Mariano Felice. 2013. Constrained grammatical error correction using statistical machine translation. In Proceedings of the 17th Conference on Computational Natural Language Learning: Shared Task. find
  25. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 http://arxiv.org/abs/1212.5701. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rei-2017-artificial-error-generation-machine-translation/