Elamur Research⌕
☰
home / papers / 2022 / Interpretability for Language Learners Using Example-Based Grammatical Error Correction

Interpretability for Language Learners Using Example-Based Grammatical Error Correction

Summary

Neural grammatical error correction (GEC) systems have become quite accurate, but they rarely tell learners why a correction was made. This leaves learners unsure whether to trust a suggestion. Kaneko et al. propose EB-GEC, which justifies each edit with a retrieved pair of incorrect and corrected sentences. It builds on kNN-MT. Decoder hidden states from the training data serve as keys, and the values are the next token plus its source and target sentences. At inference, the model mixes the usual softmax output with a distribution built from the nearest stored neighbours. The closest neighbour behind each edited token is shown to the learner as supporting evidence.

The approach works with any trained autoregressive encoder-decoder and needs no extra retrieval training. Compared with vanilla GEC, EB-GEC was more accurate on three test sets and on par on the fourth. In human evaluation, its examples were judged significantly more useful than those from earlier surface-matching retrieval baselines. The authors present this as the first evidence that retrieved examples help actual users, not just task accuracy.

Key points

  • Training data and datastore: BEA-2019 data (W&I-train, NUCLE, FCE-train, Lang-8) was used, with identical source/target pairs removed. This left about 0.6M sentence pairs, which also form the datastore. W&I-dev served as development data.
  • Evaluation: Tested on W&I-test and FCE-test (ERRANT), CoNLL2014 (M²), and JFLEG-test (GLEU).
  • Setup: Transformer-big with an 8,000-subword BPE vocabulary and beam width 5. Settings were λ = 0.5, k = 16, and temperature T = 1,000. Faiss handled the nearest-neighbour search, and results are averaged over five random seeds.
  • Edit alignment: Edits were located with Gestalt pattern matching, and one nearest example is displayed per edited token.
  • Comparison and code: The baseline example-retrieval methods were taken from the work of Arai et al. (2020), Yen et al. (2015), and Matsubara et al. (2008). Code is available at github.com/kanekomasahiro/eb-gec.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.496 · arXiv 2203.07085

@inproceedings{kaneko2022interpretability,
  title = {{Interpretability for Language Learners Using Example-Based Grammatical Error Correction}},
  author = {Kaneko, Masahiro and Takase, Sho and Niwa, Ayana and Okazaki, Naoaki},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.496},
  eprint = {2203.07085},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.496}
}

References

59 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mio Arai, Masahiro Kaneko, and Mamoru Komachi. 2019. Grammatical-error-aware incorrect example retrieval system for learners of Japanese as a second language. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 296–305, Florence, Italy. Ass… find
  2. Mio Arai, Masahiro Kaneko, and Mamoru Komachi. 2020. Example retrieval system using grammatical error detection for japanese as a second language learners. Transactions of the Japanese Society for Artificial Intelligence, 35(5):A–K23. find
  3. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 Shared Task on Grammatical Error Correction. In BEA, pages 52–75, Florence, Italy. Association for Computational Linguistics. find our card
  4. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In ACL, pages 793–805, Vancouver, Canada. Association for Computational Linguistics. find our card
  5. Yuchang Cheng and Tomoki Nagase. 2012. An example-based Japanese proofreading system for offshore development. In Proceedings of COLING 2012: Demonstration Papers, pages 67–76, Mumbai, India. find
  6. The COLING 2012 Organizing Committee. find
  7. Shamil Chollampatt and Hwee Tou Ng. 2018. A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5755–5762, New Orleans, Louisiana. Association for the Advancement of Artificial… find
  8. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better Evaluation for Grammatical Error Correction. In NAACL, pages 568–572, Montréal, Canada. Association for Computational Linguistics. find
  9. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English. In BEA, pages 22–31, Atlanta, Georgia. Association for Computational Linguistics. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Takao Doi, Hirofumi Yamamoto, and Eiichiro Sumita. 2005. Example-based machine translation using efficient sentence retrieval based on edit-distance. ACM Transactions on Asian Language Information Processing, 4(4):377–399. find
  12. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. find our card
  13. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments. In COLING, pages 825–835, Osaka, Japan. find
  14. The COLING 2016 Organizing Committee. find
  15. Sylviane Granger. 1998. Developing an Automated Writing Placement System for ESL Learners. In LEC, pages 3–18. find
  16. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  17. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909. find
  18. Rejwanul Haque, Sudip Kumar Naskar, Yanjun Ma, and Andy Way. 2009. Using supertags as source language context in SMT. In Proceedings of the 13th Annual conference of the European Association for Machine Translation, Barcelona, Spain. European Association for Machine Translation. find
  19. Kengo Hotate, Masahiro Kaneko, Satoru Katsumata, and Mamoru Komachi. 2019. Controlling grammatical error correction using word edit rate. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 149–154, Florence, Italy. Association… find
  20. Kengo Hotate, Masahiro Kaneko, and Mamoru Komachi. 2020. Generating diverse corrections with local beam search for grammatical error correction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2132–2137, Barcelona, Spain (Online). International Committee on Co… find
  21. Tim Johns. 1994. From printout to handout: Grammar and vocabulary teaching in the context of data-driven learning. Perspectives on pedagogical grammar, 293. find
  22. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547. find
  23. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  24. Megumi Makino and Kazuhide Yamamoto. 2008. Summarization by analogy: An example-based approach for news articles. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II. find
  25. Shigeki Matsubara, Yoshihide Kato, and Seiji Egawa. 2008. Escort: example sentence retrieval system as support tool for english writing. Journal of Information Processing and Management, 51(4):251–259. find
  26. Atsushi Mizumoto and Kiyomi Chujo. 2015. A meta-analysis of data-driven learning approach in the japanese efl classroom. English Corpus Studies, 22:1–18. find
  27. Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, and Mamoru Komachi. 2019. TMU transformer system using BERT for re-ranking at BEA 2019 grammatical error correction on restricted track. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 20… find
  28. Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining Revision Log of Language Learning SNS for Automated Japanese Error Correction of Second Language Learners. In IJCNLP, pages 147–155, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find
  29. Masahiro Kaneko and Mamoru Komachi. 2019. Multi-head multi-layer attention to deep language representations for grammatical error detection. Computación y Sistemas, 23(3):883–891. find
  30. Makoto Nagao. 1984. A framework of a mechanical translation between japanese and english by analogy principle. Artificial and human intelligence, pages 351–354. find
  31. Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248– 4254, Onl… find
  32. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215, Hong Kong, China.… find
  33. Masahiro Kaneko, Yuya Sakaizawa, and Mamoru Komachi. 2017. Grammatical error detection using error- and grammaticality-specific word embeddings. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 40–48, Taipei, Taiwan. Asian Fede… find
  34. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neighbor machine translation. In International Conference on Learning Representations. find
  35. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  36. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc… find
  37. Ryo Nagata, Kentaro Inui, and Shin’ichiro Ishikawa. 2020. Creating corpora for research in feedback comment generation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 340–345, Marseille, France. European Language Resources Association. find
  38. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground Truth for Grammatical Error Correction Metrics. In NAACL, pages 588– 593, Beijing, China. Association for Computational Linguistics. find
  39. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. In EACL, pages 229–234, Valencia, Spain. Association for Computational Linguistics. find
  40. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In CoNLL, pages 1–14, Baltimore, Maryland. Association for Computational Linguistics. find
  41. Kiyonori Ohtake and Kazuhide Yamamoto. 2003. Applicability analysis of corpus-derived paraphrases toward example-based paraphrasing. In Proceedings of the 17th Pacific Asia Conference on Language, Information and Computation, pages 380–391, Sen-tosa, Singapore. COLIPS PUBLICATIONS. find
  42. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. A… find
  43. Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, and Kentaro Inui. 2020. Instance-based learning of span representations: A case study through named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,… find
  44. John W Ratcliff and David E Metzener. 1988. Pattern-matching-the gestalt approach. Dr Dobbs Journal, 13(7):46. find
  45. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  46. Nicolas Stroppa, Antal van den Bosch, and Andy Way. 2007. Exploiting source similarity for SMT using context-informed features. In Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages: Papers, Skövde, Sweden. find
  47. Eiichiro Sumita and Hitoshi Iida. 1991. Experiments and prospects of example-based machine translation. In 29th Annual Meeting of the Association for Computational Linguistics, pages 185–192, Berkeley, California, USA. Association for Computational Linguistics. find
  48. Antal Van Den Bosch. 2007. A memory-based classification approach to marker-based ebmt. In Proceedings of the METIS-II Workshop on New Approaches to Machine Translation. find
  49. Maarten Van Gompel, Antal Van Den Bosch, and Peter Berck. 2009. Extending memory-based machine translation to phrases. In Proceedings of the Third Workshop on Example-Based Machine Translation, pages 79–86. find
  50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS, pages 5998–6008. Curran Asso… find
  51. Mary E Webb, Andrew Fluck, Johannes Magenheim, Joyce Malyn-Smith, Juliet Waters, Michelle De-schênes, and Jason Zagami. 2020. Machine learning for human learners: opportunities, issues, tensions and threats. Educational Technology Research and Development, pages 1–22. find
  52. Sam Wiseman and Karl Stratos. 2019. Label-agnostic sequence labeling by copying nearest neighbors. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5363–5369, Florence, Italy. Association for Computational Linguistics. find
  53. Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denoising natural language: Diverse backtranslation for grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  54. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In NAACL, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics. find
  55. Helen Yannakoudakis, Øistein E. Andersen, Geran-payeh Ardeshir, Briscoe Ted, and Nicholls Diane. 2018. Developing an Automated Writing Placement System for ESL Learners. In Applied Measurement in Education, pages 251–267. find
  56. Tzu-Hsi Yen, Jian-Cheng Wu, Jim Chang, Joanne Boisson, and Jason Chang. 2015. WriteAhead: Mining grammar patterns in corpora for assisted writing. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 139–144, Beijing, China. Association for Computational Linguistics and The Asian Federatio… find
  57. Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380–386, San Diego, California. Association for Co… find
  58. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  59. Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kaneko-2022-interpretability-language-learners-example-based/