Elamur Research⌕
☰
home / papers / 2020 / Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation

Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation

Summary

Standard metrics for grammatical error correction (GEC), such as F0.5 and GLEU, count all corrections equally. However, a spelling error is much easier to correct than a misused article or verb tense. As a result, researchers benefit more from working on frequent and simple errors. The authors propose metrics that take into account the difficulty of each error. Difficulty is determined from the outputs of the systems themselves: the larger the share of systems that correctly fixed an error, the easier it is considered. Accordingly, such an error receives a lower weight. The main technical problem is aligning the outputs of different systems when the edits include insertions and deletions. To do this, the reference sentence and each system output are split into sequences of chunks of equal length and aligned with one another. The metrics were tested on six corpora and eight systems. They rank systems more consistently than F0.5, and the difficulty estimates agree with intuition. This approach encourages correcting errors that existing systems fail to handle and makes GEC systems more diverse.

Key points

  • The weight of an error is computed as w_i = 1 − n_i/N, where n_i is the number of systems that corrected the error correctly and N is the total number of systems. The general function has hyperparameters a, b, c, but the paper studies only the simplest variant.
  • The alignment algorithm consists of several steps. First, sentences are aligned using the method from ERRANT, which minimizes the Damerau–Levenshtein distance. Then dummy chunks are added to the sequence to account for possible insertions. Next, elastic matching is applied, and the success of correcting each chunk is marked with a binary sequence.
  • The evaluation was carried out on the CoNLL-2013, CoNLL-2014, FCE, JFLEG, KJ and ICNALE corpora. The scorer and the data with difficulty weights are released as the GTS tool (github.com/gotutiyan/GTS).
  • An example for CoNLL-2014 (about 2600 errors, recall ≈ 0.45, N = 10): a single system that differs from the others must correct on its own about 130 errors (≈5%) that the other systems cannot fix in order to match them.
  • Limitations: the method requires the outputs of many systems, so it is best suited to shared tasks. The weights depend on which systems are included in the set. Multiple references inflate the weights. Correlation with expert difficulty judgments has not yet been measured.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.188

@inproceedings{gotou2020taking,
  title = {{Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation}},
  author = {Gotou, Takumi and Nagata, Ryo and Mita, Masato and Hanawa, Kazuaki},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.188},
  url = {https://doi.org/10.18653/v1/2020.coling-main.188}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adriane Boyd. 2018. Using Wikipedia edits in low resource grammatical error correction. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 79–84. find
  2. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of ACL (Volume 1: Long Papers), pages 793–805, July. find our card
  3. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better Evaluation for Grammatical Error Correction. In Proceedings of NAACL, pages 568–572. find
  4. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proc. of 26th International Conference on Computational Linguistics: Technical Papers, pages 825–835, Osaka, Japan. find
  5. Tao Ge, Furu Wei, and Ming Zhou. 2018. Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study. arXiv. find
  6. Shin’ichro Ishikawa. 2013. The ICNALE and Sophisticated Contrastive Interlanguage Analysis of Asian learners of English. Learner Corpus Studies in Asia and the World, 1:91–118. find
  7. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Phrase-based Machine Translation is State-of-the-Art for Automatic Grammatical Error Correction. In Proceedings of EMNLP, pages 1546–1556. find
  8. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task. In Proceedings of NAACL, pages 595–606. find
  9. Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of EMNLP-IJCNLP, pages 1236– 1242. find
  10. John Lee. 2004. Automatic article restoration. In Proc. of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 31–36. find
  11. Nitin Madnani, Martin Chodorow, Joel Tetreault, and Alla Rozovskaya. 2011. They can help: Using crowdsourcing to improve the evaluation of grammatical error detection systems. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pag… find
  12. Masato Mita, Tomoya Mizumoto, Masahiro Kaneko, Ryo Nagata, and Kentaro Inui. 2019. Cross-corpora evaluation and analysis of grammatical error correction models — is single-corpus evaluation enough? In Proceedings of NAACL: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1309–131… find
  13. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47. find
  14. Ryo Nagata, Edward Whittaker, and Vera Sheinman. 2011. Creating a manually error-tagged and shallow-parsed corpus. In Proceedings of ACL, pages 1210–1219. find
  15. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground Truth for Grammatical Error Correction Metrics. In Proceedings of ACL, pages 588–593. find
  16. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction. In Proceedings of EACL, pages 229–234. find
  17. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL 2013 Shared Task, pages 1–12. find
  18. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL 2014 Shared Task, pages 1–14. find
  19. Alla Rozovskaya and Dan Roth. 2019. Grammar error correction in morphologically rich languages: The case of Russian. Transactions of the Association for Computational Linguistics, 7:1–17, March. find
  20. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of ACL, pages 180–189. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gotou-2020-taking-correction-difficulty-into-account/